Thursday, June 25, 2009

eBird - Under the hood


This summer we're working on performance issues at eBird. Some birders who have lots of checklists are seeing really slow load times for things like the My eBird page, and we need to make sure that as eBird grows we can scale our tools up to meet user needs. What this means is that most of our development time will be spent on improvements that might not be obvious at first, but that should reap rewards for all users in the long run. We expect this to take a few more months.

We do have a few exciting developments that are about to be launched. The most important of which is a series of eBird APIs that feed data out to outside developers that want to make tools that use eBird data. These could include iPhone apps, Facebook apps etc. These should be launched in the coming weeks, and we're really excited about it.

We hope that you'll be patient as we go through this back-end development phase, but we do look forward to pushing out new and better tools for birders on the other side of this period.


Brian, Chris and Marshall (and Tim who is sequestered in his cube)


  1. I look forward to what you guys end up producing! *cough-xmlfeeds-cough*

  2. This is good to hear - server performance has been a real problem at times, and I'm very excited about an API. There are wonderful things we could cook up especially if the API would play well with the Google Maps API.

  3. Can't wait for feeds.

    XML! XML! XML!

  4. No glamor in the DB and performance stuff, but everybody sure notices if it's not done properly.

    Keep up the good work.

    It's so hot here, we have all the windows closed to keep the sun out. Hard to count birds with the windows closed, though.

  5. This weekend while I was out birding I thought quite a bit about several recent topics on this forum, including the "Are you submitting all identified species" thread, and the "Approvers for rare birds" thread. I wonder if it would make sense to allow the submitter to include additional information similar to what is currently done with age/sex. It would be optional (but encouraged) for all submitters, but could be required (or strongly encouraged) for rare submissions.

    I am thinking something along the lines of a dropdown/radio button for "How confident are you in this identification", maybe a 3 or 4 point scale ranging from "3: Positive identification", 2: Possible identification, positive ID to family level", "1: Potential identification, other similar species possible".

    Additionally there would be a checkbox or multi-select list of pre-coded identification methods, such as "Identified by song", "Identified visually", "Observed by 2 or more independent parties", "Photograph available", "Video available", "Only observed briefly", "Observed in flight", "Observed at nest", etc. Possibly even include a list of references used to aid Identification (guidebooks, all about birds, BNA, etc).

    To me this seems that it would be much more accurate scientifically (even if it was a common species but the observer was not positive which flycatcher that was they could denote their surety level and sources used). It would give the beginning submitter confidence in choosing yes to the "Are you submitting a complete list" question. And it could speed up or even automate the "rare bird" verification. Maybe change the site behavior to automatically "approve" any rare report that has a surety level and ID methods listed (if they are left blank, user would get an email encouraging them to submit the extra data). Users who dont submit the extra info on rare bird reports would still have the record stored and available to scientists, but would be marked "unsourced observation" and so would be less valuable to the community.

  6. You can already add additional comments to any observation you make. After submitting your counts go to MyEbird -manage my counts, select the one you want, open it up and click Edit Species and you can put all the details you want for each species. As a state reviewer I find this a very easy way to judge unusual sightings.

  7. You can add comments to the species as you enter checklist data; that way you don't have to afterwards go back into your account and modify.

    My usual routine is to:

    1) Enter the numerical data for each species.

    2) Go back to the top and enter any "rare birds" in the box provided; usually a non-species level ID, e.g.: "accipiter sp."

    3) Then check the button that I want to add age/sex data. Here I add whatever species level comments I want. (If you do this before step two, then step two erases species comments.)

    4) Put in any general comments at the checklist bottom.
    5) Submit.

  8. but if the comments can be encoded rather than free-text, then they can be used directly by researchers to retrieve a recordset with highly specific requirements.

    For instance it would be a highly manual task (and is, from the prior postings regarding the backlog for some reviewers) to read and interpret each submitter comment separately. But if its coded, the need for reviewers is greatly reduced (as is the natural variation that will always occur between the different reviewers) since the researchers could directly request a recordset checked via a simple query customized for their specific project needs (eg only "ivory-billed woodpecker" sightings that had "confirmed identification", "multiple independent parties confirmed identification" and "has video" or "has photo").

    Even the fieldbook editors could use the data, for instance if people using Sibley's had more/less confidence in ID'ing a downy woodpecker (ie vs a hairy WP) as compared to people using Peterson's etc etc

  9. But would submitters really take all the extra time to click on all those extra radio buttons/drop-down menus? I got exhausted just reading the 4 paragraph suggestion response. It made me think ebird would probably lose a lot of submitters if it had that many extra things to submit. And when you're birding every day and know your birds really well, it's going to get redundant hitting "3" for every single species.

    I do however like the idea of submitting how one observed/identified the birds. I find myself occasionally doing this in the comments, especially if I saw an individual nesting.

  10. All

    The key to making this kind of thing workable is to make these additional use preferences. That way beginners can keep the streamlined version of eBird as is, and the advanced birders can do all the clicking of radio buttons that they like. I like these ideas, and we'll certainly knock them around when we get to developing eBird again.



  11. Can't wait for the Iphone app!