{"API Performance"}

To Incentivize API Performance, Load, And Security Testing, Providers Should Reduce Bandwidth And Compute Costs Asscociated

I love that AWS is baking monitoring testing by default in the new Amazon API Gateway. I am also seeing new service from AWS, and Google providing security and testing services for your APIs, and other infrastructure. It just makes sense for cloud platforms to incentivize security of their platforms, but also ensure wider success through the performance and load testing of APIs as well.

As I'm reading through recent releases, and posts, I'm thinking about the growth in monitoring, testing, and performance services targeting APIs, and the convergence with a growth in the number of approaches to API virtualization, and what containers are doing to the API space. I feel like Amazon baking in monitoring and testing into API deployment and management because it is in their best interest, but is also something I think providers could go even further when it comes to investment in this area.

What if you could establish a stage of your operations, such as QA, or maybe production testing, and the compute and bandwidth costs associated with operations in these stages were significantly discounted? Kind of like the difference in storage levels between Amazon S3 and Glacier, but designed specifically to encourage monitoring, testing, and performance on API deployments.

Maybe AWS is already doing this and I've missed it. Regardless it seems like an interesting way that any API service provider could encourage customers to deliver better quality APIs, as well as help give a boost to the overall API testing, monitoring, and performance layer of the sector. #JustAThought

See The Full Blog Post


The New Mind Control APIs That Salesforce Is Testing On Conference Attendees Is Available To Premier Partners

The Dreamforce conference is happening this week in San Francisco, a flagship event for the Platform as a Service (PaaS) company. Salesforce is one of the original pioneers in API technology, allowing companies to empower their sales force, using the the latest in technology. Something that in 2015, Salesforce is taking this to the next level, with a handful of attendees, and partners in attendance at the conference.

Using smart pillow technology, Salesforce will be testing out a new set of subliminal mind control APIs. All attendees of the Dreamforce conference have agreed to be part of the tests, through their acceptance of the event terms of service, but only a small group of 500 individuals will actually be targeted. Exactly which attendees are selected will be a secret, even from the handful of 25 partners who will be involved in the test. 

Through carefully placed hotel pillows, targeted attendees will receive subliminal messages, transmitted via smart pillow APIs developed by Salesforce. Messages will be crafted in association with partners, testing out concepts of directing attendees what they will eat the next day, which sessions they are attending, where they will be going in the exhibit hall, and who they will be networking with The objective is to better understand how open the conference attendees are open to suggestion, in an conference environment.

While some partners of this mind control trial are just doing random tests to see if the technology works, others are looking to implement tasks that are in sync with their sales objectives. Ernst Stavro Blofeld, CEO of Next Generation Staffing Inc, says "the Salesforce test represents the future of industry, and the workforce--this weeks test is about seeing what we can accomplish at a conference, but represents what we will be able to achieve in our workforce on a daily basis."

Salesforce reminded us that this is just a simple test, but an important one that reflects the influence the company already has over its constituents. The company enjoys one of the most loyal base of business users, out of all leading software companies in the world, and this new approach to targeting a loyal base of users, is just the beginning of a new generation of API engineered influence.

See The Full Blog Post


Testing New Publishing

This is my testing.

See The Full Blog Post


Introducing Runscope Metrics: API Performance and Usage Analytics


Some of your most popular feature requests have been for reporting and analytics -- specifically performance (latency) and usage (consumption) metrics. Today, we're announcing the release of Runscope Metrics for all customers. The first two Runscope Metrics reports that we're releasing are Performance and Usage. Runscope Radar is very useful for testing if an API is operating properly. You may have noticed that while defining tests, an assertion for "Response Time (ms)" is an option. True, that catching a failing test is important (i.e. not responding within time threshold), but tracking performance data on successful tests is just as important. Gradual increases in latency (response time) can be signs of a backend service that is not scaling well. Spikes of latency can be an indication of intermittent network problems that are unrelated to the health of the backend service.  Both of these cases are common and mostly go undetected.  These spikes and gradual increases in latency are now easily spotted with visual performance graphs. By catching latency issues early, developers can investigate and address them before they grow into major problems. Keeping tabs on the number of API requests an app makes is also important. Modern APIs implement rate limits and throttles — for example, restricting apps to X calls per hour, or Y calls per day. In most cases, an app that exceeds a limit is denied access which could lead to application failure. It's unfortunate, but that's usually what it takes for developers to discover their API request capacity problem. Using Runscope Metrics, developers can stay several steps ahead by monitoring their usage. All API calls that proxy through Runscope get logged. Similar to the latency performance report, the usage report makes it easy for developers to spot both gradual trends in growth as well as spikes. This report helps developers to forecast usage trends and plan accordingly. The default reporting view for both Performance and Usage Reports is across all hosts. Finding the exact method that is experiencing latency issues, or ramping up on API call consumption, is very easy. Developers can refine the scope of each report by simply clicking a hostname from the list. From the hostname view, drilling down to the endpoint path and method is done exactly the same way. Runscope Metrics is available to all customers. If you need help understanding Runscope Metrics reports, send us a note. Our support team is standing by, ready to help.

URL: http://blog.runscope.com/posts/introducing-runscope-metrics-api-performance-and-usage-analytics

See The Full Blog Post


Contributing To The Testing & Monitoring Lifecycle

Contributing To The Testing & Monitoring Lifecycle

When it comes to testing, and monitoring an API, you begin to really see how machine readable API definitions can be the truth, in the contract between API provider and consumer. API definitions are being used by API testing and monitoring services like SmartBear, providing a central set of rules that can ensure your APIs deliver as promised.

Making sure all your APIs operate as expected, and just like generating up to date documentation, you can ensure the entire surface area of your API is tested, and operating as intended. Test driven development (TDD) is becoming common practice for API development, and API definitions will play an increasing role in this side of API operations.

An API definition provides a central truth, that can be used by API providers to monitor API operations, but also give the same set of rules to external API monitoring services, as well as individual API consumers. Monitoring, and understanding an API up time, from multiple external sources is becoming a part of how the API economy is stabilizing itself, and API definitions provide a portable template, that can be used across all API monitoring services.

Testing and monitoring of vital resources that applications depend on is becoming the norm, with new service providers emerging to assist in this area, and large technology companies like Google, making testing and monitoring default in all platform operations. Without a set of instructions that describe the API surface area, it will be cumbersome, and costly, to generate the automated testing and monitoring jobs necessary to produce a stable, API economy.

See The Full Blog Post


APITools Raises The Bar With Open, On-Premise API Testing and Monitoring Tools

APITools, the cloud-based API integration services is raising the bar for the space by introducing an open source, on-premise version of their API monitoring service. APITools only launched this year, and because of consumer demands, they moved up the timeframe for open sourcing the platform, which was already on the roadmap.

I’d say after API design, API integration services and tooling, for testing, monitoring, and transforming API calls is one of the fastest growing segments of the API space. We are seeing solid solutions from SmartBear, Runscope, TheRightAPI, Nomos Software, and from API pioneer John Musser, with API Science, but APITools is definitely raising the stakes with open sourcing theirs offering.

The world of API integration service and tooling is rapidly expanding, and only time will tell whether developers prefer running in the cloud, or on-premise, and what features they are looking for, something I've been documenting as I study what each of the companies offer.

I suspect, that along with other API design, deployment, and management tools we'll need a mix of freemium, open tiers, on-premise, and enterprise API integration services and tooling, to meet the demands of this fast growing segment that overlaps with both providing and consuming APIs.

Disclosure: APITools is a 3Scale service, and 3Scale is an API Evangelist partner.

See The Full Blog Post


Beta Testing Linkrot.js On API Evangelist

I started beta testing a new JavaScript library, combined with API, that I’m calling linkrot.js. My goal is to address link rot across my blogs. There are two main reasons links are bad on my site, either I moved the page or resource, or a website or other resource has gone away.

To help address this problem, I wrote a simple JavaScript file that lives in the footer of my blog, and when the page loads, it spiders all the links on the page, combining them into a single list and then makes a call to the linkrot.js API.

All new links will get a URL shortener applied, as well as a screenshot taken of the page. Every night a script will run to check the HTTP status of each link used in my site—verifying the page exists, and is a valid link.

Every time link rot.js loads, it will spider the links available in the page, sync with linkrot.js API, and the API returns the corresponding shortened URL, or if a link shows a 404 status, the link will no longer link to page, it will popup the last screenshot of the page, identifying the page no longer exists.

Eventually I will be developing a dashboard, allowing me to manage the link rot across my websites, make suggestions on links I can fix, provides a visual screen capture of those I cannot, while also adding a new analytics layer by implementing shortened URLs.

Linkrot.js is just an internal tool I’m developing in private beta. Once I get up and running, Audrey will beta test, and we’ll see where it goes from there. Who knows!

See The Full Blog Post


API Testing and Monitoring Finding A Home In Your Companies Existing QA Process

I've been doing API Evangelist for three years now, a world where selling APIs to existing companies outside of Silicon Valley, and often venture capital firms is a serious challenge. While APis have been around for a while in many different forms, this new, more open and collaborative approach to APis seems very foreign, new and scary for some companies and investors--resulting in them often very resistant to it.

As part of my storytelling process, I'm always looking for ways to dovetail API tools and services into existing business needs and operations, making them much more palatable to companies across many business sectors. Once part of the API space I'm just getting a handle on is the area API integration, which includes testing, monitoring, debugging, scheduling, authentication and other key challenges developers face when building applications that depend on APIs.

I was having a great conversation with Roger Guess of TheRightAPI the other day, which I try to do regularly. We are always brainstorming ideas on where the space is going and the best way to tell stories around API integration, that will resonate with existing companies. Roger was talking about the success they are finding dovetailing their testing, monitoring and other web API integration services with a company's existing QA process--something that I can see will resonate with many companies.

Hopefully your company already has a full developed QA cycle for your development team(s), including, but not limited to, automated, unit and regression testing--something where API tests, monitoring, scheduling and other emerging API integration building blocks will fit in nicely. This new breed of APi integration tools don't have to be some entirely new approach to development, chances are you are already using APIs in your development and API testing and monitoring can just be added to your existing QA toolbox.

I will spend more time looking for stories that help relate some of these new approaches to your existing QA processes, hopefully finding news ways you can put tools and services like TheRightAPI to use, helping you better manage the API integration aspect of your web and mobile application development.

See The Full Blog Post


Netflix API Is Much More Than A Public API

Netflix has entered the final stages of shuttering its public API last week. Its been coming for a while now, starting in June of 2012, and now is official with the platform no longer accepting new API registrations.

After reading about the changes to the Netflix Public API program on their blog, and hearing much of the news in response, everyone seems to file this away, along with the Twitter API--just another API platform screwing over its developers.

As I do, I wanted to take a step back, look at the bigger picture and try understand what happened.  On October 1st 2008, Netflix launched their public API, and they appear to have done everything right. They had a blog, solicited code samples from developers, accepted application submissions and even showcased the developers apps in the gallery. Netflix would even help promote your app to Netflix subscribers and threw hackathons. The Netflix API team worked to improve API performance, communicate regularly, but really nothing that amazing happened.

There were applications like InstaWatcher and WhichFlicks (among others) developed on the API, but as Daniel Jacobson puts it, a thousand flowers didn’t bloom. In these situations its easy to blame the API provider, but developers didn’t really step up and build anything that innovative and cool. So is this a failure of Netflix? A failure of developers to innovate? Or could it possibly be a third: failure of the API vision?

I would say the demise of the Netflix public API is equal part Netflix and the developer, and just the nature of the industry it exists in. It didn’t take me long to look through the Netflix API blog, so I can tell they didn’t put alot into evangelizing the API. But I really can’t find any innovation that occurred by developers as part of it, so I think us devs have to share some of the responsibility as well.

Several of the blog posts covering the news last week, compared this to Twitter which I think for the untrained eye of the mainstream tech blogosphere, this is easy to do. But Twitter is user generated content, via one of the newest types of content platforms, and Netflix is heavily licensed and policed content from one of the oldest content platforms. I think expecting public API success from Netflix and / or developers was a lot to ask.

I love and believe in APIs, but I’m not delusional enough to think they will work magically everywhere they are applied. However, even with the closing of the public Netflix API, I consider Netflix is an API success story. Look what they’ve done with their internal and partner APIs. They’ve managed to scale not just from the data center to the cloud, but globally and across 800+ devices--while also sharing this knowledge and wisdom with the public via their blog:

If that wasn't enough, they are also open sourcing much of the technology behind their approach:

  • eureka - AWS Service registry for resilient mid-tier load balancing and failover
  • RxJava - a library for composing asynchronous and event-based programs using observable sequences for the Java VM
  • Governator - A library of extensions and utilities that enhance Google Guice to provide: classpath scanning and automatic binding, lifecycle management, configuration to field mapping, field validation and parallelized object warmup
  • Priam - Co-Process for backup/recovery, Token Management, and Centralized Configuration management for Cassandra
  • edda - Service to track changes in your cloud recipes-rss - RSS Reader Recipes that uses several of the Netflix OSS components
  • astyanax - Cassandra Java Client
  • karyon - The nucleus or the base container for Applications and Services built using the NetflixOSS ecosystem
  • netflix-graph - Compact in-memory representation of directed graph data
  • asgard - Web interface for application deployments and cloud management in Amazon Web Services (AWS)
  • Hystrix - Hystrix is a latency and fault tolerance library designed to isolate points of access to remote systems, services and 3rd party libraries, stop cascading failure and enable resilience in complex distributed systems where failure is inevitable
  • servo - Netflix Application Monitoring Library
  • frigga - Utilities for working with Asgard named objects

When measuring the success or failures of API initatives, we can't use the same yardstick in all scenarios. When you look at the knowledge, wisdom and code that has come out of Netflix, there is no way you can say their API initiative is anything but a success. I don’t see see Netflix as a case study in how to stream movies over the web via public APIs, but a deeply important experiment in how to deliver licensed content to over 800 devices, via the next generation of APIs.  Something that probably isn't an edge case, it actually represents where we all might be headed in the near future.

Let’s not get caught up in the recent deprecation of the Netflix public API.  There is so much going on!  Let's get studying some of the knowledge and technology coming out of Netflix. I know its my motivation for writing this post, and doing this research.

See The Full Blog Post


From ETL to API Reciprocity, Looking at 20 Service Providers

I spent time this week looking at 20, of what I’m calling API reciprocity providers, who are providing a new generation of what is historically known as ETL in the enterprise, to connect, transfer, transform and push data and content between the cloud services we are increasingly growing dependent on.

With more and more of our lives existing in the cloud and via mobile devices, the need to migrate data and content between services will only grow more urgent. While ETL has all the necessary tools to accomplish the job, the cloud democratized IT resources, and the same will occur to ETL, making these tools accessible by the masses.

There are quite a few ETL solutions, but I feel there are 3 solutions that are starting to make a migration towards an easier to understand and implement vision of ETL:

 

These providers are more robust, and provide much of the classic ETL tools the enterprise is used to, but also have the new emphasis on API driven services. But there are 10 new service providers I’m calling reciprocity platforms, that demonstrate the potential with offering very simple tasks, triggers and actions that can provide interaction between two or more API services:

I consider reciprocity an evolution of ETL, because of three significant approaches:

  • Simplicity - Simple, meaningful connections with transfer and tranformations that are meaningful to end users, not just a wide array of ETL building blocks an IT architect has to implement
  • API - Reciprocity platforms expose meaningful connections users have the cloud services they depend on. While you can still migrate from databases or file locations as with classic ETL, reciprocity platforms focus on APIs, while maintaining the value for end-users as well as the originating or target platforms
  • Value - Reciprocity focus on not just transmitting data and content, but identifying the value of the payload itself and the relationships, and emotions in play between users and the platforms they depend on

This new generation of ETL providers began the migration online with Yahoo Pipes. Which resonated with the alpha developers looking to harvest, migrate, merge, mashup and push data from RSS, XML, JSON and other popular API sources--except Yahoo lacked the simplicity necessary for wider audience appeal.

While I feel the 10 reciprocity providers isted above represent this new wave, there are six others incumbents trying to solve the same problem:

While studying the approach of these 20 reciprocity providers, it can be tough to identify a set of common identifiers to refer to the value created.  Each provider has their own approach and potentially identifying terminology. For my understanding, I wanted to try and establish a common way to describe how reciprocity providers are redefining ETL.  While imperfect, it will give me a common language to use, while also being a constant work in progress.

For most reciprocity providers, it starts with some ecompassing wrapper in the form of an assembly which describes the overall recipe, formula or wrapper that contains all the moving ETL parts.

Within this assembly, you can execute on workflows, usually in a single flow, but with some of the providers you can daisy chain together multiple (or endless) workflows to create a complex series of processes.

Each workflow has a defining trigger which determines the criteria that will start the workflow such as new RSS post or new tweet, and with each trigger comes a resulting action which is the target of the workflow, publishing the RSS post to a syndicated blog or adds the tweet to a Google Spreadsheet or Evernote, or any other combination of trigger and action a user desires.

Triggers and actions represent the emotional connections that are the underpinnings of ETL’s evolution into a more meaningful, reciprocation of value that is emerging in the clouds. These new providers are connecting to the classic lineup of ETL interfaces to get things done:

  • Databases
  • Files
  • Messaging
  • Web Service

While also providing the opportunity for development of open connectors to connect to any custom database, file, messages and web services. But these connectors are not described in boring IT terms, they are wrapped in the emotion and meaning derived from the cloud service--which could have different meanings for different users. This is where one part of the promise of reciprocity comes into play, by empowering average problem owners and every day users to define and execute against these types of API driven agreements.

All of these actions, tasks, formulas, jobs or other types of process require the ability to plan, execute and audit the processes, with providers offering:

  • Scheduling
  • History / Logging
  • Monitoring

With data being the lifeblood of much of these efforts, of course we will see “big data” specific tools as well:

  • Synchronization
  • Data Quality
  • Big Data
  • Analytics

While many reciprocity providers are offering interoperability between two specific services, moving data and resource from point a to b, others are bringing in classic ETL transformations:

  • Reformat
  • Aggregate
  • Sort
  • Dedupe
  • Filter
  • Partition
  • Merge
  • Join
  • Split
  • Convert

After the trigger and before the action, there is also an opportunity for other things to happen, with providers offering:

  • Push
  • Events

During trigger, action or transformation there are plenty of opportunities for custom scripting and transofrmations, with several approaches to custom programming:

  • Custom Scripts
  • JavaScript
  • Command Line
  • API

In some cases the reciprocity provider also provides a key value store allowing the storage of user specified data extracted from trigger or action connections or during the transformation process. Introducing a kind of memory store during the reciprocal cycle.

With the migration of critical resources, many of the leading providers are offering tools for testing the process before live execution:

  • Test
  • Debugger
  • Sandbox
  • Production

With any number of tasks or jobs in motion, users will need to understand whether the whole apparatus is working, with platforms offering tools for:

  • Performance
  • Monitoring
  • Optimization

While there are a couple providers offering completely open source solutions, there are also several providing OEM or white label solutions, which allow you to deploy a reciprocity platform for your partners, clients or other situations that would require it to branded in a custom way.

One area that will continue to push ETL into this new category of reciprocity providers is security. Connectors will often use OAuth, respecting a users already established relationship with platform either on the trigger or action sides, ensureing their existing relationship is upheld. Beyond this providers are offering SSL to provide secure transmissions, but in the near future we will see other layers emerge to keep agreements in tact, private and maintain the value of not just the payload but the relationships between platforms, users and reciprocity providers.

Even though reciprocity providers focus on the migration of resources in this new API driven, cloud-based world, several of them still offer dual solutions for deploying solutions in both environments:

  • Cloud
  • On-Premise

There is not one approach either in the cloud, or on premise that will work for everyone and all their needs. Some data will be perfectly find moving around the cloud, while others will require a more sensitive on-premise approach. It will be up to problem owners to decide.

Many of this new breed of providers are in beta and pricing  isn’t available. A handful have begun to apply cloud based pricing models, but most are still trying to understand the value of this new service and what market will bear. So far I’m seeing pricing based upon:

  • Seat
  • Assembly
  • Tasks
  • Connections
  • Extension
  • Sync
  • Support
  • Training

Much like IaaS, PaaS SaaS and now BaaS, reciprocity providers will have a lot of education and communication with end users before they’ll fully understand what they can charge for their services--forcing them to continue to define and differentiate themselves in 2013.

One of the most important evolutionary areas, I’m only seeing with one or two providers, is a marketplace where reciprocity platform users can browse and search for assemblies, connectors and tasks that are created by 3rd party providers for specific reciprocity approaches. A marketplace will prove to be how reciprocity platforms serve the long tail and niches that will exist within the next generation of ETL. Marketplaces will provide a way for developers to build solutions that meet specific needs, allowing them to monetize their skills and domain expertise, while also bringing in revenue to platform owners.

I understand this is all a lot of information. If you are still ready this, you most likely either already understand this space, or like me, feel it is an important area to understand and help educate people about. Just like with API service providers and BaaS, I will continue to write about my research here, while providing more refined materials as Github repos for each research area.

Let me know anything I'm missing or your opinions on the concept of API reciprocity.

See The Full Blog Post


75 Features From Across 31 BaaS Providers

I’m currently tracking on 31 backend as a service providers, in an effort to better understand how this new breed of platforms are helping developers build web and mobile apps. After looking at all the BaaS providers, there are 13 clear leaders:

Then there are another 18 other players, trying to play catch up in a space that is working hard to define itself in 2013:

My goal is to better understand what features are offered across these 31 BaaS providers. To accomplish this, I spent no more than an hour per provider looking through their sites and playing with their products to get at least a basic understanding of their offerings.

When looking for features I tried to standardize the best I could, but it is difficult when there are different approaches to the deployment of resources on each platform. I found about 75 distinct features being offered across the 31 BaaS providers. I’m sure there are other features, and vital details missing, but I wanted to start somewhere. Here is what I found, organized as best I could:

User Management

  • User
  • User Roles
  • LDAP

Content Management System (CMS)

Data

  • Table
  • Relational
  • Key Value
  • Browser
  • MySQL Connector
  • PostGres Connector
  • Oracle Connector
  • Caching
  • XML
  • CSV

File Management

  • Storage
  • Sync

Image & Photo Management

  • Storage
  • Gallery & Collections
  • Processing

Custom Code / Objects

Programmatic Interfaces

  • Web Service Connectors
  • REST API
  • Custom REST API
  • Query

Commerce

  • Product Catalog
  • Shopping Cart

Virtual Commerce

  • In-App Purchases
  • Custom Virtual Store 
  • Virtual Goods Management 
  • Currency Maintenance 
  • Virtual Economy Regulation

Other Monetization

  • Promotions
  • Subscriptions
  • Billing
  • Passbook

Ranking

  • Recomendations
  • Reviews
  • Ratings
  • Likes

Advertising

Communication

  • SMS
  • Email
  • Email Templates
  • Push Notification
  • Interactive Voice Response (IVR)
  • Messaging System

Calendar Events

Posts

Friends

Shared Links

Geo

  • Spatial
  • Location
  • Check-In
  • Places

Gaming

  • Players
  • Ranking
  • Scores
  • Boards
  • State

3rd Party Integration

  • Twitter
  • Facebook
  • Dropbox
  • Fitbit
  • Foursquare
  • Github
  • Instagram
  • LinkedIn
  • Meetup
  • Tumblr
  • Withings
  • Wordpress
  • Yammer
  • Twilio
  • Underscore
  • SendGrid
  • Moment
  • Mandrill
  • Mailgun
  • CrowdFlower
  • Google Places
  • Google Apps
  • Salesforce
  • SAP
  • Siebel
  • Wordpress

SSL

Availability

  • Performance
  • Scaling
  • Load Balance

Deployment

  • On-Premise
  • Virtual Private Cloud
  • Public Cloud

Environment

  • Sandbox
  • Production

Utility

  • Logging
  • Backups
  • Clients
  • Jobs

Analytics

These BaaS providers support a wide variety of mobile devices, platforms, frameworks in multiple languages:

Mobile Devices

  • iOS
  • Android
  • Windows
  • Blackberry

Reader Devices

  • Kindle

Mobile Platforms

  • PhoneGap
  • Trigger.io
  • Titanium

App Frameworks

  • ql.io

Automation

  • Temboo

Languages

  • JavaScript
  • Java
  • C#
  • PHP
  • Python
  • Ruby

There were many different ways the BaaS platforms provided support to its developers:

Support

  • Phone
  • Web
  • Chat
  • Dedicated Account
  • Dedicated Tech

I found 10 different ways that BaaS providers delivered pricing:

Pricing

  • API Calls
  • Push Notification
  • Bandwidth
  • Storage
  • Active Users
  • Analitics
  • Support
  • App
  • Synchronization
  • Features

Marketplace

You can view all 75 features at the BaaS Github Repository I setup. Let me know any that you feel are missing, and I’ll consider adding.

Next up, I will add the features into my BaaS tracking database and publish a breakdown of providers, with the features they offer. Letting people search and filter, and also open up to each BaaS provider to comment and submit additional features they offer.

See The Full Blog Post


Digital Strategy: 20 Federal Agencies, 76 data API and 75 Mobile API Initiatives

It has been a while since I provided an update on the White House Digital Strategy.  I monitor the progress of federal agencies participation programmatically, using JSON reports published by each agency at the agencies domain, /digitalstrategy.

After running the script today, I notice 20 federal agencies with active footprints.  There are about five more, but there are issues with the JSON version of their digital stratgies--I really want to focus on the programmatic value. So, across these 20 agencies I find 76 data API initiatives and 75 mobile API projects.

To bring you up to speed, there are two specific milestones in the Digital Government Strategy that specifically address API deployment:

  • 2.2 (Data) - Make high-value data and content in at least existing two major customer-facing systems available through web APIs, apply metadata tagging and publish a plan to transition additional high-value systems
  • 7.2 (Mobile) - Optimize at least two existing priority customer-facing services for mobile use and publish a plan for improving additional existing services  

To help shed light on where these progressive agencies are going with both their data and mobile intiatives I wanted to break them out into separate groups, for each agency.

National Science Foundation (NSF)

2.2 Data

  • Scientists and Engineers Statistical Data System (SESTAT) - The Scientists and Engineers Statistical Data System (SESTAT) provides longitudinal information on the education and employment of the college-educated U.S. science and engineering workforce, collected through three biennial surveys. These surveys capture trends in employment opportunities and salaries in various degree fields, and help federal researchers to evaluate the effectiveness of equal opportunity efforts.
  • NSF Award Information - Information pertaining to NSF awards from 1959 through the present. Data includes principal investigator, awardee institution, NSF program and associated NSF organizations, award amount, award dates, award abstract, and publications and conference proceedings produced as a result of the research.
  • NSF Research Grant Funding Rates - NSF funding rates for competitive research proposals by organizational unit. (Funding rates constitute the number of awards divided by the number of actions for a given year by organizational unit).
  • Science & Engineering Indicators - Science and Engineering Indicators (SEI) provide U.S. and international data on the following subjects: 1) Elementary and Secondary Mathematics and Science Education 2) Higher Education in Science and Engineering 3) Science and Engineering Labor Force 4) Research and Development: National Trends and International Comparisons 5) Academic Research and Development 6) Industry, Technology, and the Global Marketplace 7) Science and Technology: Public Attitudes and Understanding 8) State Indicators.
  • Project Outcome Reports - Section 7010 of the America COMPETES Act requires that researchers funded in whole or in part by NSF report on the outcomes of the funded research for the general public. Project Outcomes Reports describe the project outcomes or findings, addressing the intellectual merit and broader impacts of the work as defined in NSF merit review criteria.
  • Graduate Research Fellowship Program Awardees and Honorable Mention Recipients - Demographic information for recipients of NSF GRFP awards. The GRFP provides three years of graduate education support for individuals who have demonstrated the potential for significant achievements in science and engineering research

7.2. Mobile

  • News - NSF press releases and other agency news products
  • Funding - Catalog of NSF’s funding opportunities with links to Grants.gov System
  • Discoveries - Articles describing notable results from NSF funded research
  • Awards - Searchable database of NSF awards.
  • Staff Directory - Senior management list, organization list, browse-able and searchable staff list.
  • Events - Calendar of NSF events.
  • Vacancies - Current job vacancies at NSF.
  • Directions - Directions for visiting NSF headquarters in Ballston.
Department of State

2.2 Data

  • Bibliographical Metadata of the Foreign Relations of the United States Series - Raw bibliographical metadata for the nearly 500 official historical documentary volumes of U.S. foreign policy in the Foreign Relations of the United States published since 1861
  • aoprals.state.gov - Foreign Travel Per Diem rates
  • J1 Visa Exchange Visitor Program website - Provide accessible, plain-language information on the J1 visa Exchange Visitor Program
  • ForeignAssistance.gov - The goal of the Foreign Assistance Dashboard is to make all U.S. Government foreign assistance investments available in an accessible and easy-to-understand format.
  • Travel.State.gov - Provides an informational portal into Consular Affairs pages on international travel, passports, visa, international child abduction, and international law and policy.
  • U.S. Passport Issuance Data - U.S. Passports Issued per Fiscal Year (2010-1996); U.S. Passports Issued per Calendar Year (1995-1974); U.S. Passport Applications Received by Fiscal Year (1986-2010); Passport Issuance by State per Fiscal Year (2007-2010)

7.2. Mobile

  • Bibliographical Metadata of the Foreign Relations of the United States Series - Raw bibliographical metadata for the nearly 500 official historical documentary volumes of U.S. foreign policy in the Foreign Relations of the United States published since 1861
  • aoprals.state.gov - Foreign Travel Per Diem rates
  • Foreign Service Mobile App - The goal of this app is to serve as a learning tool to educate diverse university students (undergraduate and graduate), alumni and mid-career professionals about the various career opportunities in the Foreign Service and provide the information and resources to help them prepare for the selection and hiring processes.
  • Usembassy.gov (450+ websites under the usembassy/usconsulate.gov domain) - Provides public affairs/diplomacy information and critical citizen services (passports, visa, and business opportunities) for in-country American citizens.
  • J1 Visa Exchange Visitor Program website - Provide accessible, plain-language information on the J1 visa Exchange Visitor Program
  • ForeignAssistance.gov - The goal of the Foreign Assistance Dashboard is to make all U.S. Government foreign assistance investments available in an accessible and easy-to-understand format.
  • U.S. Passport Issuance Data - U.S. Passports Issued per Fiscal Year (2010-1996); U.S. Passports Issued per Calendar Year (1995-1974); U.S. Passport Applications Received by Fiscal Year (1986-2010); Passport Issuance by State per Fiscal Year (2007-2010)
Department of Housing and Urban Development (HUD)

2.2 Data

  • HUD User - HUD USER provides interested researchers with access to the original data sets generated by PD&R-sponsored data collection efforts, including the American Housing Survey, HUD median family income limits, as well as microdata from research initiatives on topics such as housing discrimination, the HUD-insured multifamily housing stock, and the public housing population.
  • Low Rent Apartment Search - The government gives funds directly to apartment owners, who lower the rents they charge low-income tenants. You can find low-rent apartments for senior citizens and people with disabilities, as well as for families and individuals.
  • Fair Market Rents - The Fair market rents (FMR) and Income limits data app will provide users the ability to easily obtain statistics for FMR and income limits by their present or other locations.
  • PD&R Edge - Provides access to PD&R's on-line magazine "the Edge". TheEdgeis an online magazine providing news, a message from the Assistant Secretary, and a wide range of information on housing and community development issues, regulations, and research that is updated biweekly.The Edgeis available on Apple iOS and Android powered smartphones.ThePD&R EdgeApp directs users to the PD&R mobile webpage where up- to -date content can be accessed. The app allows users to share content on social media networks like Facebook and Twitter, and via email.
  • Public Housing (PHA) Contact - View contact information for Public Housing Agencies in your city and state
  • Enterprise GIS - Data is a key component to any location-based application. Enterprise GIS aims to remove data acquisition, integration, and maintenance obstacles for geo-developers by providing access to premium HUD data sets.

7.2. Mobile

  • Housing Counselor - HUD sponsors housing counseling agencies throughout the country that can provide advice on buying a home, renting, defaults, foreclosures, and credit issues. This app allows you to select a list of agencies for each state below. You may search more specifically for areverse mortgage counseloror if you are facing foreclosure, search for aforeclosure avoidance counselor.
  • File a Fair Housing Complaint - Federal law prohibits housing discrimination based on your race, color, national origin, religion, sex, familial status, or disability. By creating a mobile app for the HUD Form 903, FHEO will expand the ability to reach anyone that feels that their civil rights have been violated and have them submit a fair housing complaint easily and confidentially.
  • FHEO HUD.gov mobile adaptive web content - Expanding access to Fair Housing an
  • Housing Discrimination Investigative Checklist - Developing and updating the design of the FHEO investigative checklist that is used by FHEO investigative resources.
  • HUDMaps - HUD has developed a number of Geospatial Information Systems (GIS) that are currently available on the web and are listed at http://egis.hud.gov/. The HUDMap tool allows HUD Employees and Contractors to pull information from various internal and external sources to assist programmatic and disaster response projects. HUDMaps access is being expanded to provide for mobile device capabilities.
  • GMP Monitoring Exhibits Handbook - The CPD Monitoring Handbook includes all programs and technical functions for which CPD Field staff have monitoring responsibilities.
Federal Energy Regulatory Commission (FERC)

2.2 Data

  • Decisions and Notices - Provides a monthly collection of Delegated Orders, Notices, and Commission Decisions from Commission Meetings or Notational Voting arranged by date.
  • eTariff - Allows for tariffs, tariff revisions and rate change applications to be filed electronically in the manner prescribed by Order No. 714. The affected regulated entities are: • Public utilities and Power Marketing Administrations under Parts 35 and 300; • Natural gas pipelines under Parts 154 and 284; • Intrastate gas pipelines under Part 284; and • Oil pipelines under Part 341.
  • eLibrary - eLibrary is a records information system that contains: 1. Electronic versions of documents issued by FERC from 1989-Present; 2. Documents received and issued by FERC: a. A description of documents from 1981-Present; b. Microfilm and aperture cards of documents for 1981-1995; c. Scanned images of paper documents from 1995-Present; and d. Native files electronically submitted from November 2000-Present
  • eService - Provides users with official mailing list or service list for a docketed proceeding.
  • Electric Quarterly Reports (EQR) - Allows users to access data submitted by utilities and power marketers. Access is achieved through the following means: 1. Download Spreadsheets - Contract and transaction data by company by quarter in a structure similar to the one used for data import. 2. Summary Reports - Short summaries of each company's EQR filings (beginning Q1 2005) identifying the products they sell, the customers they sell to and the control areas where deliveries are made. 3. Filing Inquiries - EQR data can be retrieved using standard queries which can be customized by the user. 4. Selective Filings Download - Retrieves data on multiple companies and quarters with one request. Processed overnight and sent via email the next day. 5. Download Database - Provides for the download of the full EQR database. It should be used only by advanced users with several gigabytes of disk space.

7.2. Mobile

  • Decisions and Notices - Provides a monthly collection of Delegated Orders, Notices, and Commission Decisions from Commission Meetings or Notational Voting arranged by date.
  • What’s New - The “What’s New” RSS feed provides news and information details about the events at FERC.
  • eSubscription - Users subscribe or ‘sign up’ for specific dockets and are notified via email about future correspondence. Users have immediate access to the correspondence or documents in eLibrary.
  • eFiling - Allows users to electronically submit qualified documents to FERC in lieu of paper filings.
  • eRegistration - eRegistration provides the FERC customer an easy-to-use entry point to do business with all FERC Online applications. Think of eRegistration as a form of membership. By registering, the user will receive a single user id and password that allows them to transact all of their business with FERC. eRegistration is valuable to any person who transacts business with the FERC on behalf of themselves or another organization (e.g. companies or corporations). It provides authentication support to the FERC Online applications that ensures safe and secure transactions, thereby protecting the integrity of your data.
  • Electric Quarterly Reports (EQR) - Allows all public utilities and power marketers to file EQRs for the most recent calendar quarter. The filings must summarize contractual terms and conditions for: Market-based power sales, Cost-based power sales, and Transmission service.
National Archives and Records Administration (NARA)

2.2 Data

  • FederalRegister.gov - Integration of the Regulations.gov API into FederalRegister.gov and its API. This integration would provide greater access to public comments and supporting documents in Regulations.gov, and improve process for submitting public comments from FederalRegister.gov into Regulations.gov.
  • FederalRegiser.gov API - Expand the FederalRegister.gov API to include the \"Public Inspection Desk.\"
  • Code of Federal Regulations; Federal Register - Develop an API for FDsys through the Office of Federal Register-Government Printing Office Partnership
  • National Archives Catalog on Wikipedia - Make additional National Archives records available through Wikipedia, which is accessible through the MediaWiki API
  • National Archives Catalog on Flickr - Make additional National Archives records available on Flickr, which is accessible through the Flickr API

7.2. Mobile

  • FederalRegister.gov - Mobile optimize FederalRegister.gov
  • Daily Compilation of Presidential Documents - Develop a mobile application based on the Daily Compilation of Presidential Documents
  • Archives.gov - Mobile optimize Archives.gov
  • National Archives Catalog on Wikipedia - Make additional National Archives records available through Wikipedia, which is mobile optimized and available through mobile apps
  • National Archives Catalog on Flickr - Make additional National Archives records available through Flickr, which is mobile optimized and available through mobile apps
Social Security Administration (SSA)

2.2 Data

  • SSA State Agency Monthly Workload Data - Monthly information from October 2000 onwards concerning claims for disability benefits that were referred for a disability determination to one of the 54 state agencies. The data may be used to examine disability application filing trends by time and by state, state agency workloads, and disability claims outcomes.
  • Average Wait Time Until Hearing Held Report - A presentation of the average time (in months) from the hearing request date until a hearing is held for claims pending in the Office of Disability.
  • Hearing Office Workload Data - A monthly presentation of four key workload indicators (pending, receipts, dispositions and average processing time) for each hearing office in the Office of Disability Adjudication and Review (ODAR).
  • Hearing Office Average Processing Time Ranking Report - A monthly ranking of the 165 ODAR hearing offices (including 3 satellite offices) by the average number of days until final disposition of the hearing request. The average shown will be a combined average for all cases completed in that hearing office. The public will be able to determine where a particular hearing office stands among the total with respect to this workload category.
  • SSA/Department of State Identity Verification Web Service - SSA is providing the DOS with information to verify the identity of passport applicants via a web service. This is being done based on a reimbursable agreement with DOS.

7.2. Mobile

  • Mobile Supplemental Security Income (SSI) Wage Reporting Application - SSI recipients will use this application to report their monthly wage amounts. This enables them to meet their monthly reporting obligations.
  • Mobile Optimized Frequently Asked Questions (FAQs) - People seeking information from SSA will use this mobile optimized FAQ website to obtain information about numerous topics.
  • Mobile Contact - People seeking to do business with SSA will use this application to get needed information and directions to their local SSA office. In addition they will have access to information about services available by phone and online.
  • Mobile Optimized Life Expectancy Calculator - The Life Expectancy Calculator is a valuable financial planning tool which we encourage the public to use to help decide when to retire and begin collecting Social Security benefits. The Calculator uses the gender and birth date entered by the user to provide the average number of additional years a person of the same gender and age can expect to live when he reaches a specific age. Providing a mobile optimized version of the Calculator also provides links to other helpful retirement planning tools and helps to promote other online services.
Department of Agriculture (USDA)

2.2 Data

  • National Farmers Market Directory - AMS produced director containing information about U.S. farmers market locations, directions, operating times, product offerings, and accepted forms of payment. Supports local and regional food systems, as well as development of local economies.
  • Office Information Profile System - USDA Service Centers are designed to be a single location where customers can access the services provided by the Farm Service Agency, Natural Resources Conservation Service, and the Rural Development agencies. This tool provides the address of a USDA Service Center and other Agency offices serving your area along with information on how to contact them.
  • ERS Charts of Note - Highlights from our current and past research currently provided in daily Charts of Note available via Web APIs (Application Programming Interfaces, a system of machine-to-machine interactions over a network), and an embeddable widget code
  • Digital Asset Management - A DAM system (DAMS) is a combination of software, hardware and professional services that provides a central location for storing, managing and accessing digital assets - both the files and the accompanying metadata. Digital assets include images, graphics, logos, animations, audio/video clips, presentations, pages, documents and a number of other digital file formats.

7.2. Mobile

  • USDA Newsroom - The USDA Newsroom holds official news releases, statements, transcripts and speeches released by the Department.
  • USDA Blog - The Blog features content from all USDA agencies and features the latest news, events and features. The Blog also provides the public an opportunity to ask questions or share their thoughts about the latest issues.
  • AmberWaves eZine - Fully mobile-optimized online magazine, and magazine app for offline reading on tablets and other mobile devices
  • LDP/PCP Rates - Mobile Version of the Loan Deficiency Payment and Posted County Price rate system
  • Meat and Poultry Inspection Directory - The Meat, Poultry and Egg Product Inspection Directory is a listing of establishments that produce meat, poultry, and/or egg products regulated by USDA's Food Safety and Inspection Service (FSIS) pursuant to the Federal Meat Inspection Act, the Poultry Products Inspection Act, and the Egg Products Inspection Act.
Department of Education (ED)

2.2 Data

  • EDFacts - The purpose of EDFacts is to collect and report K-12 education performance data for use by policymakers and Department of Education program offices. With relevant, actionable data supplied by EDFacts, decision-makers can identify which programs are working, have insight into education progress at the state and district levels, identify gaps and best practices, and make sound education policy and budgetary decisions to improve education outcomes. http://www2.ed.gov/about/inits/ed/edfacts/index.html
  • State Education Data Profiles - Web-based search tool for statewide information in elementary/secondary education, postsecondary education and selected demographics for all states in the U.S. using a variety of NCES data sources including NAEP, IPEDS, and CCD data. http://nces.ed.gov/programs/stateprofiles/
  • College Navigator - A web-based tool for searching all colleges and universities in the United States. College Navigator consists primarily of the latest data from the Integrated Postsecondary Education Data System (IPEDS), the core postsecondary education data collection program for the National Center for Education Statistics. http://nces.ed.gov/collegenavigator/
  • Program Information Publication System (Part of Program Information on the Web) - One of ED's core purposes is to inform the public about funding opportunities (programs). Currently program information is available on the Web in HTML web pages. ED is developing a central database repository (Program Information Publication System or PIPS) with a web API. ED.gov will call on the PIPS web API to publish program information on the website and provide an online search tool. Customers will be able to find more consistent, up-to-date program information quickly and easily. Information for this initiative is limited to programs authorized and funded under federal law as well as other related efforts, and encompasses several ED offices.

7.2. Mobile

  • G5 Grants Management System - G5 supports the Agency's grant making business process and is a full lifecycle end-to-end grants management system (from intake of applications, peer review, award, payment, performance monitoring and final closeout of the grant award. https://www.g5.gov
  • StudentAid.gov - StudentAid.gov is the first step in a multi-phase project to provide consumers with a one-stop website where they can access federal student aid information, apply for federal aid, repay student loans and navigate the college decision-making process. Whether you're a student, a parent, a borrower in repayment, an educator or a professional engaged in influencing and informing students and borrowers, StudentAid.gov has useful information for you. The site, available in English and Spanish, combines content and interactive tools from several U.S. Department of Education websites and makes it easy for you to find the information you need. It also features videos and infographics to help answer the most frequently asked questions about financial aid accessible via smartphones and tablets. http://www.studentaid.gov
  • Homeroom Blog - Homeroom Blog is the official blog of the Department. The purpose of these blog is to facilitate an ongoing dialogue on education issues.
  • College Navigator - Web-based tool for searching all colleges and universities in the United States. College Navigator consists primarily of the latest data from the Integrated Postsecondary Education Data System (IPEDS), the core postsecondary education data collection program for the National Center for Education Statistics. http://nces.ed.gov/collegenavigator/
  • ED.gov - ED.gov is the Department's primary Internet portal and website. As such, it performs the following functions to ensure efficient and expanded public access and communication between the government and citizens: convey the Department's brand and key messages, provide relevant and timely information, hosts sites for ED offices and programs, and provide a unified entry point to other ED resources. http://www.ed.gov/
Department of Justice (DOJ)

2.2 Data

  • ATF Trace Data Report - A key component of the Bureau of Alcohol, Tobacco, Firearms and Explosives’ (ATF) enforcement mission is the tracing of firearms on behalf of thousands of Federal, State, local and foreign law enforcement agencies. Firearms trace data is critically important information developed by ATF. ATF has prepared the following state-by-state reports utilizing trace data which is intended to provide the public with insight into firearms recoveries.
  • National Sex Offender Website and Database - The Dru Sjodin National Sex Offender Public Website (NSOPW), coordinated by the U.S. Department of Justice, is a cooperative effort between jurisdictions hosting public sex offender registries (“Jurisdictions”) and the federal government and is offered free of charge to the public. These Jurisdictions include the 50 states, U.S. Territories, the District of Columbia, and participating tribes. The Website provides an advanced search tool that allows a user to submit a single national query to obtain information about sex offenders; a listing of public registry Web sites by state, territory, and tribe; and information on sexual abuse education and prevention.
  • FOIA.gov - FOIA.gov houses annual Freedom of Information Act (FOIA) data from all agencies subject to the FOIA. This data is collected annually and available to the public as PDF reports or in various machine-readable formats on FOIA.gov.
  • Uniform Crime Report - The Uniform Crime Reporting (UCR) Program was conceived in 1929 by the International Association of Chiefs of Police to meet a need for reliable, uniform crime statistics for the nation. In 1930, the FBI was tasked with collecting, publishing, and archiving those statistics. Today, several annual statistical publications are produced from data provided by nearly 17,000 law enforcement agencies across the United States.
  • National Crime Victimization Survey - NCVS is the Nation's primary source of information on criminal victimization. Each year, data are obtained from a nationally representative sample of about 40,000 households comprising nearly 75,000 persons on the frequency, characteristics and consequences of criminal victimization in the United States. Each household is interviewed twice during the year. The survey enables BJS to estimate the likelihood of victimization by rape, sexual assault, robbery, assault, theft, household burglary, and motor vehicle theft for the population as a whole as well as for segments of the population such as women, the elderly, members of various racial groups, city dwellers, or other groups. The NCVS provides the largest national forum for victims to describe the impact of crime and characteristics of violent offenders.

7.2. Mobile

  • Justice.gov - The main website of the Department of Justice
  • StopFraud.gov - The website of the President's Financial Fraud Task Force hosting information about the work of the task force as well as resources to about fraud - including prevention tips and where to report crimes if they occur.
  • Civil Rights Division Report a Violation Web Resources - The Civil Rights Division enforces civil rights laws in a wide variety of contexts. This resource directs individuals on how to submit a complaint or report of a potential civil rights violation.
  • Office on Violence Against Women Resource Map - A comprehensive list of national, state, local and tribal resources for victims of violence against women.
Department of Energy (DOE)

2.2 Data

  • ALTERNATIVE FUELING STATION LOCATION DATA - API - The underlying data used to populate the Alternative Fueling Station Locator tool is available via a Web service. This data set is considered the most trusted industry resource for location-based alternative fueling station data because of the long-standing collection process fostered by relationships with industry and fuel providers and its continuous and rigorous vetting process. Using this API, developers can access the data to build their own mobile apps, widgets, or tools. Developers may choose to mash up the station data set with numerous other data sets available on the Web to create useful products to fill industry-specific needs. http://developer.nrel.gov/doc/transportation
  • DOE GREEN ENERGY - This API data service provides green energy results from research and development conducted throughout the Department and by DOE-funded awards at universities. The service allows extraction of two data sets: green energy technical (approximately 40,000) reports and green energy patents (over 2,000). http://www.osti.gov/GreenEnergyXMLManual.pdf
  • Electricity API - The U.S. Energy Information Administration (EIA) collects, analyzes, and disseminates independent and impartial energy information to promote sound policymaking, efficient markets, and public understanding of energy and its interaction with the economy and the environment. The Electricity API project will make this dataset publically available.Beta Site: http://www.eia.gov/beta/api/
  • OpenEI - OpenEI continues to strive to be a global leader in open-data source for energy information--specifically analyses on renewable energy and energy efficiency. The platform is a Wiki, similar to Wikipedias Wiki, which many users are already familiar with and include an API. Users can view, edit, and add data, " and download data for free. https://www.openei.org

7.2. Mobile

  • MOBILE SITE FOR JOBS AT THE ENERGY DEPARTMENT - The site features a searchable data pull from USAJobs of the currently available Energy Department job opportunities. It is updated daily, allowing prospective employees to search for the latest opportunities wherever and whenever they want. In addition to the job search capabilities, there are resources available to the public concerning student, veteran, and other office programs/opportunities.
  • ENERGY.GOV - The Energy Department’s main site, Energy.gov and its contents, are now available on the go. This allows users to access the Energy Departments’ resources over a variety of mobile devices such as smart phones and tablets. The American public is becoming increasingly mobile and the Energy department is responding to this demand
  • ENERGY CAREER GAME - This is a fun, fast-paced puzzle/strategy game intended to generate interest and recruit the next generation of individuals and entrepreneurs in the energy industry. In the game, the player is tasked with managing resource networks between cities, power plants, homes, and businesses to provide power to these buildings. The game will educate players about the unique challenges and economic opportunities faced in meeting America’s energy needs and transitioning to a clean energy future. It will give players a better understanding of the career opportunities in the energy sector as well as the Energy Department. The game is being developed for the Department free of charge by college students who are working with the Department’s Office of Human Capital. The game will also incorporate a plug-in that will allow players to post their scores to Facebook. Expected to be ready for beta release by October 2012.
National Aeronautics and Space Administration (NASA)

2.2 Data

  • NASA Data API - The data.nasa.gov API allows a machine-readable interface to return metadata from the site organized by category, tag, date, or search term. We’re hoping this allows new and creative visualizations of the data resources NASA provides to the public. Additionally, it is a learning experience for us as we work to expand transparency, participation, and collaboration at NASA through new uses of technology.
  • ISS Live API - Thousands of data points are downloaded every minute from the Station, and ISS Live! makes a broad set of that data open and accessible. Furthermore, ISS Live! will make an application programming interface (API) available as a web service for external developers to take ISS data and put it into their own websites and mobile applications.
  • ExoAPI - ExoAPI is an ongoing project that extends the accessibility of exoplanetary data by providing an easy to use RESTful API. ExoAPI was created during the NASA Space Apps Challenge by a team of three amazing geniuses who knew nothing about space before they started this...and still don't really. Currently the data is provided by http://exoplanetology.blogspot.com/ who in turn feeds the data from http://exoplanet.eu/. The ExoAPI team plans on extending the API to encompass a wider array of data sources and more interesting space data to reach as many programmers as possible and encourage an explosion of space data based mashups.

7.2. Mobile

  • NASA Apps Store - Provide employees and contractors with access to agency data and systems on the go.
  • WebTADS - WebTADS Mobile is a lighter version of the desktop-based WebTADS developed to provide NASA Civil Servants with the convenience of recording time when they're not in the office or connected via VPN.
  • Visualization Explorer - NASA Visualization Explorer, the coolest way to get stories about advanced space-based research delivered right to your iPad. A direct connection to NASA’s extraordinary fleet of research spacecraft, this app presents cutting edge research stories in an engaging and exciting format.
  • people.nasa.gov - NASA Enterprise Directory (NED) Search
Nuclear Regulatory Commission (NRC)

2.2 Data

  • Reactor Operating Status Reports - A web-based system that allows the NRC to report on the operating status and power output of commercial power reactors located within the continental United States
  • Power Reactor Daily Event Reports - A web-based system that allows the NRC to report on the daily events and activities occurring at commercial power reactors located within the continental United States
  • Operating Reactor Inspection Reports - A web-based system that allows the NRC to report on observations and findings of inspections occurring at commercial power reactors located within the continental United States
  •   - A web-based system that allows the NRC to report on licensed facilities, activities, or basic component that fail to comply with NRC regulations
  • Part 21 (Component Defect Reports) - A web-based system that allows the NRC to communicate with the public daily on our regulatory activities and events

7.2. Mobile

  • Public Meeting Feedback System - A web-based system that allows the NRC to collect public meeting feedback from meeting attendees
  • Top 5 Most-Accessed Agency Web Pages - A web-based system that allows the NRC to communicate with the public on our regulatory activities and events
United States Agency for International Development (USAID)

2.2 Data

  • USAID Portfolio Map - Map depicts the locations of USAID-funded projects to better monitor development results, improve aid effectiveness and coordination, and enhance transparency and social accountability. The map depicts the total number of unique projects at the administrative one boundary level (i.e. region, state, province). For example, a Global Health project may operate in several provinces within a country, in which case it will be given a count of one in each of those provinces. A project count will not be given to a province if that project does not operate there. National projects are depicted across all administrative one boundaries to show the impact of where the work is taking place.
  • Famine Early Warning System Network (FEWS NET) - A USAID-funded food security and famine early warning system covering more than 30 of the most food insecure countries in the world
  • U.S. Overseas Loans and Grants (Greenbook) - These data are U.S economic and military assistance by country from 1946 to 2010. This is the authoritative data set of U.S. foreign assistance. The data set is used to report U.S foreign assistance to Congress as required by the Foreign Assistance Act, Section 634.

7.2. Mobile

  • Development Experience Clearinghouse - USAID's Development Experience Clearinghouse (DEC) is the largest online resource for USAID funded technical and program documentation, with over 141700 documents available for electronic download. Search USAID's online database of agency-funded technical and program-related documents to download USAID documents in PDF format for free.
  • USAID Portfolio Map - The Portfolio Map is a mobile app for accessing information about the development work USAID is performing every day. The app will give mobile device users the ability to browse our portfolio for a subset of the countries in which USAID is working. The app will provide general country overviews at a glance and also will allow users to access more detailed information as needed.
Department of Defense (DOD)

2.2 Data

  • United States Army Corps of Engineers (USACE) Lockage Data - The Corps Locks website contains lock and vessel specific information derived from the United States Army Corps of Engineers Lock Performance Monitoring System (LPMS).
  • United States Army Corps of Engineers (USACE) Commodity Data - The USACE Commodity system provides commodities information to the Department of Agriculture on a weekly basis.

7.2. Mobile

  • TRICARE Website - Provides TRICARE Coverage Plan information to active and retired Uniformed Services and their families
  • Defense Commissary Agency (DeCA) Commissaries Locator - Provides users the closet store hours, directions, contact information, floor plans, etc.
  • Arlington National Cemetery Grave Site Locator - Enables veterans, family members and the public to locate gravesites; generate front and back photos of a headstone or monument; and receive directions to those locations.
Department of the Treasury

2.2 Data

  • Wholesale Securities Services (WSS) - Provides information on recent auction results (issue date, discount rate, investment rate, offer amount), upcoming auctions (auction date, issue date, offer amount), and savings bond rates.
  • Summary Debt Accounting Services - Gives users the ability to find the total public debt outstanding for a specific day or date range. The data components will be useful to feed the Federal Reserve System's FRED.
  • OCC Derviatives and Capital Markets Quarterly Reports - Each quarter, based on information from the Reports of Condition and Income (call reports) filed by all insured U.S. commercial banks and trust companies as well as other published financial data, the Office of the Comptroller of the Currency prepares a report. That report describes what the call report information discloses about banks' derivative activities

7.2. Mobile

  • IRS2Go (English) - A smartphone application that lets you interact with the IRS using your mobile device. IRS continues innovation with its award-winning mobile application, IRS2Go. The new version will have improved access to IRS’s online services and may include additional features such as include “Pay your tax bill”, “Learn about your notice”, “Find a form”, “Find an eFile provider”, “Find a practitioner”, “Find a VITA Site nearest you”.
Department of Commerce

2.2 Data

  • Export.gov API - An API that will allow developers access to Export.gov\\\'s events data
  • Census API - Additional economic and demographic data

7.2. Mobile

  • Commerce.gov - Ensuring mobile-optimization of Commerce.gov
  • Census mobile apps - Census will develop 2 additional mobile apps
Department of Health and Human Services (HHS)

2.2 Data

  • HealthCare Finder API (http://healthdata.gov/data/dataset/healthcare-finder-api) - All of the data used on the Finder.HealthCare.gov web application is available through the API. There are multiple collections of data available through the API including Public Options Data, Individual and Family Health Insurance Options Data, and Small Group Insurance Options Data. Visit http://www.hhs.gov/digitalstrategy/open-data/index.html to learn more about this API.
  • HealthData.gov Catalog API (http://healthdata.gov/catalog-api) - The HealthData.gov API is used to provide software developers with programmatic access to the contents of our data catalog. The API can be used to find recently added datasets, to search the catalog, to download the contents of the catalog for analysis, or to build a new data catalog tool. Visit http://www.healthdata.gov/catalog-api to learn more about this API.

7.2. Mobile

  • Medicare.gov (http://www.medicare.gov/) - Medicare.gov is the consumer website for Medicare beneficiaries, caregivers, and advocates. The implementation of responsive design on this site allows us to support traditional desktop PCs, tablets, and smartphones all from one URL and code base. Learn more about the redesign process here: http://www.hhs.gov/digitalstrategy/mobile/medicare-responsive-design.html We have made the responsive design code available to the public here: http://www.hhs.gov/digitalstrategy/blog/2012/10/medicare-assets.html
  • HHS Digital Strategy Site (http://www.hhs.gov/digitalstrategy) - HHS uses this website to engage the public, report progress on the implementation of the Digital Government Strategy, showcase digital strategy best practices, and test new technology and tools. Visit http://www.hhs.gov/digitalstrategy/ to learn more.
Department of Transportation (DOT)

2.2 Data

  • Federal Railroad Administration 10 Year Accident Reports - The 10-Year Accident/Incident Report, one of the most requested reports on the Federal Railroad Administration’s website, provides historical statistics on rail-related train accidents, injuries and fatalities, highway-rail crossing collisions and operational information. We will create an API to render all this information in a user-interactive dashboard program.
  • SaferCar.gov Recall and Complaint Data - The National Highway Transportation Safety Administration will make vehicle safety data available in API format. Specifically, we will create an API with safety-related complaints about motor vehicles and motor vehicle equipment, including vehicle make, model and year. We will also create an API with recall data for vehicle products that have safety-related defects or do not comply with federal motor vehicle safety standards.

7.2. Mobile

  • SaferCar App - DOT will create a user-friendly iOS smartphone product to streamline access to information on SaferCar.gov, including vehicle 5-star safety ratings; vehicle defects; and the Child Safety Seat Locator. The app will also allow consumers to search and submit vehicle complaints
  • Grade Crossing App - The Federal Railroad Administration is developing an application that will provide users with mobile access to Grade Crossing Information. Users will be able to visualize the rail grade crossings on a map interface, find their own location on the map, and view crossing specific information such as Accident Data and Inventory Reports from FRA’s safety data website.
Federal Communications Commission (FCC)

2.2 Data

  • Broadcast Public Inspection File - For decades, the public file for each station has been kept at the station’s main studio in paper form (or more recently in electronic form at some stations) and made available during normal business hours. Relying on the advantages of current technology, however, the Commission has now changed its rules to require almost all of this public file information for television stations to be posted online at this site.
  • License View - Spectrum is a national resource. License View provides information on over 3 million FCC issued licenses for use of the nation's airwaves and other purposes.

7.2. Mobile

  • fcc.gov - The entire fcc.gov experience will be a mobile ready application this fall.
  • National Broadband map - The National Broadband Map (NBM) is a searchable and interactive website that allows users to view broadband availability across every neighborhood in the United States. The NBM was created by the National Telecommunications and Information Administration (NTIA), in collaboration with the Federal Communications Commission (FCC), and in partnership with 50 states, five territories and the District of Columbia. The NBM is part of NTIA's State Broadband Initiative. The NBM is updated approximately every six months and was first published on February 17, 2011.
Office of Personnel Management (OPM)

2.2 Data

  • USAJOBS - Website to post and find federal jobs. By providing an API for USAJOBS, we will make it easier for third parties to (1) provide customized access to job postings (e.g., only those that would be of interest to members of a particular professional organization) or (2) combine job posting with other services or applications (e.g., discussion boards or other social media around federal jobs).
  • Federal Government Operating Status - Provides information on the operating status of federal agencies. Not only do federal employees need to know the operating status of their offices, but also many businesses and other organizations base their operating status decisions on ours. The API for the operating status will make it easier for those organizations and even other federal agencies to make full use of operating status data.

7.2. Mobile

  • USAJOBS - Website to post and find federal jobs. By keeping our popular iOS mobile app up-to-date and creating an Android version, we will better serve more people seeking federal employment.
  • Federal Operating Status - Provides information on the operating status of federal agencies. Not only do federal employees use this information whenever there is bad weather or an unexpected major event such as an earthquake, but local businesses and other organizations often make their own operating status decisions based on the federal government’s decision. Although OPM has never created an application for the operating status, one exists for iOS devices and is sold by a private entity for $0.99. We will make this freely available information also more freely accessible with a mobile application for both iOS and Android devices.

See The Full Blog Post


The Battle for Your API Proxy

Every Web API is designed to receive requests from and respond to the outside world.

Every day an API can receive thousands or potentially millions of calls. Before the API can process these requests and returns a response, it has to potentially tackle a huge laundry list of functionalities:
  • Identity / Authentication
  • Traffic Controls
  • Rate Limiting
  • Performance
  • Security
  • Scalability
  • Filtering
  • Encryption
  • Logging
Once all these items are handled, then the API can do what it is designed to do -- process its payload and return a response.

Many API owners tackle all these layers of the API themselves. But there are also several service providers out their looking to do this for them.

The first group of API service providers in this area use what I call the Proxy Flow Through, and this includes Mashery and Apigee. Mashery and Apigee deliver these service by routing all calls to an API through their proxy. Each call actually is made to Mashery and Apigee, then they route the request to the actual API for a response.

The second group of API service providers in this area, use what I call the Proxy Connector, and this includes 3Scale and Mashape. 3Scale and Mashape deliver these services by providing a connector your API can use to communicate with the proxy during each call.

All of these service providers end up delivering a similar set of services. This proxy, whether flow through or connector tackle the needs listed above, but then also provide much needed data on API operations. Apigee, Mashery, and 3Scale provide you with tools for monitoring and analyzing this operational data.

There is a lot at stake here. The next wave of Internet growth will pass through these proxies. More API owners are turning to these providers to deliver this layer, where all usage of the API from web apps, mobile apps, and other devices developed internally, by partners and the public will pass through these proxies.

These proxies are becoming brokers in the API economy. Each service provider is competing, for the ability to proxy your web API and be your broker in this new economy.

Mashery, Apigee, and 3Scale all provide some very robust services via their proxy, Apigee even offers their proxy technology as an on-premise appliance.

Newcomer Mashape does not provide a full suite of services via their proxy, like the others, but Mashape is taking a different approach, by introducing an add-on layer within their proxy. There are only two add-ons available (billing, rate limit) currently, but this concept opens up an entirely new type of marketplace for the entire API industry, not just a single API. Developers can now build specialized tools to sell, and push forward whats possible within a proxy. This proxy add-on layer also has potential for allowing a more a la carte set of services available to API owners.

The demand for API proxies is growing, and each service provider is pushing the definition of what it means to proxy your API. Mashery, Apigee, 3Scale, Mashape and others are working hard to define the space, push it forward, while also winning market-share.

One might compare this playing field to the competition between database service providers of the last two decades, where companies like Microsoft and Oracle battled it out for market share. There are many differences for sure, but with the amount of information and value running through APIs, the parallels to database choices are there -- except I still don't see the MySQL of API proxies as of yet.

See The Full Blog Post


Internet Explorer 9 - HTML 5 Support

See The Full Blog Post


Amazon EC2 Volume Types and Performance Testing

I have not had much time to do performance testing on our recent Amazon EC2 environment. We got our entire company migrated and stabilized in a completely Amazon Web Services based environment. And so far I love it..

So I am very thankful I came across this performance testing of different types of file system configurations setup on Amazon volumes.

Checkout the Amazon Performance Testing here.

It is very helpful to have other considering using Amazon Web Services and EC2 as an alternative to their current ISP or hosting provider. Helps me make better educated decisions with the limited time and resources I have.

See The Full Blog Post