The Bases of Data

"He has the most doubles of any right-handed batter in history." It was a claim confidently bandied about as Craig Biggio was considered for the National Baseball Hall of Fame. And it seemed an objectively true and relatively simple sort of fact — the number of doubles he had hit was greater than the number of doubles any other right-handed player had hit. It was easy to check by heading to baseball-reference.com or some other encyclopedia.

Baseball Reference's list of most doubles hit in a career includes Tris Speaker, Pete Rose, Stan Musial, and Ty Cobb — all either switch- or left-handed-hitters — and then Craig Biggio, with 668 career doubles. But how do we know that Biggio really hit 11 more doubles than the next right-handed hitter on the list, Nap Lajoie? Lajoie played from 1896 to 1916, before sabermetrics, fantasy leagues, and highlight reels — even before radio broadcasts or daily statistical updates. More troubling, at the time of Biggio's candidacy, the powers that be in Major League Baseball disagreed with Baseball Reference and other organizations about whether Lajoie or Cobb had the highest batting average in 1910. Given such distance and uncertainty, how can we be confident Lajoie didn't have more doubles lurking in the records — or that he actually did hit exactly 30 doubles in, say, 1907?

Even a simple claim about performance statistics implies a reliable record of hits going back nearly 150 years. It may seem an objective fact, right or wrong, but that's not to suggest it is a simple or easy thing to be confident about. Believing that Biggio set a record for doubles requires believing in an entire history of recordkeeping and error checking, in an entire structure of people and tools meant to ensure the accuracy and reliability of facts. If we want to figure out how we know that Lajoie hit 30 doubles in 1907, then we might as well start by asking where Baseball Reference actually got that number.

Baseball Reference's clean interface makes the facts displayed there seem natural, eternal, and indisputable. Biggio's page reveals a dizzying array of numbers, sorted neatly by year and category. Some of the stats provided, like "batting average" and "runs scored," are essentially as old as professional baseball; others, such as "wins above replacement" and "adjusted batting runs," are more recent creations. The interface provides a clever bubble that appears when the cursor is hovered over a statistic, explaining how the number has been calculated. The site even allows users to sum across seasons or other subcategories. The whole structure is geared toward providing a clear display of mathematical certainty. Or, as the founder of the site, Sean Forman, explained, the site's purpose is to "answer questions as quickly, easily, and accurately as possible."

When Forman first put his website online in mid-2000, its ability to generate quick answers was its selling point. Even in this relatively early stage of the internet, there were already other places where fans could find similar data online, including stats.com and totalbaseball.com. These competitors often also had big names, or at least names with authority — totalbaseball.com had signed agreements with Sports Illustrated, and stats.com was licensed by a variety of national publications.

The advantage baseball-reference.com offered was a superior interface, which Forman called putting a "friendly face" on existing data. He minimized images and ads, with 95 percent of the pages under 20 kilobytes (kb) — no minor thing, given residential download speeds generally maxed out at 56 kb per second at the turn of the century. Forman had started Baseball Reference as he was finishing his doctoral dissertation on computational protein folding, a field seemingly irrelevant to baseball until he explains that his research was "basically optimization." Forman was good at taking a complicated mess of facts and interconnections, analyzing them, and cleaning them up.

The casual fan might assume Baseball Reference's numbers were coming directly from Major League Baseball or from its official statistician, the Elias Sports Bureau. At the time of Biggio's candidacy, however, Forman had no formal relationship with Elias, and he had never spoken with anyone there. As is the case with many encyclopedias, the specific origins of any given statistic, save a generic note at the bottom of every page, were left unspecified. However elegant its interface, Baseball Reference didn't — and doesn't — provide many overt reasons to trust the statistics that appear there.

As it turns out, Forman had initially taken his data from the statistical database freely provided online in 1996 by another internet-savvy baseball fan, Sean Lahman, at baseball.com. Lahman, in turn, had built his database using the CD-ROM that came along with the third edition of the groundbreaking encyclopedia Total Baseball in 1993. The CD included image files of the entire encyclopedia, with its own reader on the disk to view the individual files. Lahman noticed that the publishers of the CD, Creative Multi-media Corporation, didn't protect its contents very well. With a day job designing databases of digital images for Kodak, Lahman had the skills to post Total Baseball's statistics online for anyone to download.

It's misleading to talk about posting statistics online as if Lahman were simply copying the files from the CD-ROM. A book is just as much a technology for holding and displaying data as a computer file — and perhaps has proved more robust and user-friendly. But Lahman didn't just want to read the book on a computer. Lahman wanted to reverse-engineer a database. He gathered ("scraped") the statistics from the files and then organized them into a relational database by assigning unique IDs to each player, team, and statistical category so that they would be easily searchable. Ultimately, he was able to create his own database, one that relied on the facts as conveyed by Total Baseball but that was presented not as the image of a printed table, but as an editable Microsoft Access file.

Lahman decided to put his database online as a result of two frustrations: first, that so many online repositories unexpectedly disappeared in the early days of the internet, and second, that baseball data were often presented in ways that were not conducive to research. Watching Ken Burns's film Baseball during the 1994 strike had given Lahman the idea of combining his computing skills with his interest in baseball. In this period before it was common to "surf" the "world wide web," many connoisseurs of baseball statistics had found their way to an active Usenet group: rec.sports.baseball. Lahman found the baseball group a useful resource for sharing statistical research and interesting facts among fans who were also early adopters of computing technology.

However, Lahman soon discovered that it was extremely in-efficient to conduct independent research in order to respond to Usenet posts. While it was easy enough for someone to find the answer to basic questions such as the number of doubles Lajoie hit in 1907, it was not easy to find how many doubles had been hit that year by left-handed second basemen or how many doubles had been hit by second basemen over a particular span of years, unless that question had been specifically addressed by an encyclopedia's authors. Someone would have to go through the books player by player, team by team, and year by year. Turning Total Baseball into a downloadable database would make Lahman's and other researchers' lives easier: now they could do their own searches and manipulations of the data. The advantage of even rudimentary computerized databases was that they made it possible to query information without needing to collect and reorganize the data every time. A few commands could select all second basemen in 1907, aggregate their doubles totals, and display the results in a matter of seconds. Lahman's database transformed the way baseball research could be done.

In turn, Forman's Baseball Reference was revolutionary because it eliminated any need for skill in database manipulation. By downloading Lahman's database and then constructing a website that would break it up into linked pages, Forman made it possible for people who wanted to know about doubles hit in 1907 to access the data directly through his website. The trick for Forman was writing code such that he could take requests from nonprogrammers, turn them into formal queries on his own server, and then send out the relevant page of data to display on a user's home computer. He could do this by 2000 in a way that Lahman couldn't, or that at least would have been much harder, a few years earlier because an "off the shelf" database management application, MySQL, had not been available until the last half of the 1990s.

Both Lahman and Forman took existing statistics and figured out new ways to display them. If they did not create the data themselves, were they violating copyright by displaying it as their own? It certainly wasn't a farfetched possibility. Before Lahman posted Total Baseball's statistics online, he had consulted an intellectual property lawyer to get the go-ahead first.

Of course, Total Baseball itself had "taken" data from previous publications. There was a tradition extending back more than a century of repackaging and reprinting historical statistics from the Spalding, Beadle, DeWitt, and Reach guides, as well as from the series of titles produced by the Sporting News. Who owned the right to print the fact that Lajoie hit 30 doubles during the 1907 season?

As the U.S. Supreme Court stated in 1991, the problem of assigning ownership to data is the "undeniable tension" between "two well-established propositions. The first is that facts are not copyrightable; the other, that compilations of facts generally are." The 1991 case in question was Feist Publications, Inc. v. Rural Telephone Service. Both parties were publishers of telephone books — making their dispute a bit dated — and their conflict surrounded the fact that Feist Publications had reprinted over 1,000 listings from Rural's directory in its own book without Rural's assent. The Court found that compilations of facts, like telephone books or statistical tables, are subject to copyright, but not because of the "sweat of the brow," the work that went into the creation of facts. "The distinction," the Court clarified, "is one between creation and discovery: the first person to find and report a particular fact has not created the fact; he or she has merely discovered its existence." Copyright was not a "reward" for the work of compiling facts. Rather, it was an acknowledgement of the creativity and originality of the presentation of facts. In this case, since Rural was not found to have had any originality in its presentation of the facts (it had simply listed entries alphabetically, failing to display the requisite "minimal creative spark"), Feist was not considered to have violated copyright law by reprinting thousands of entries.

On these terms, Lahman and Forman could reprint the data from Total Baseball because the facts themselves couldn't be copyrighted. Both Lahman's database and Forman's website took existing facts and packaged them in new and creative ways. The physical presentation of the facts was what mattered. The statistics themselves were treated simply as part of the natural world, separable from any effort that might have gone into their "discovery."

* * *

Pete Palmer certainly had expended "sweat of the brow" in compiling the statistics that went into the Total Baseball CD-ROM in 1993. Palmer had spent much of his youth looking through baseball guides and creating lists from them. He made the kinds of lists curious kids were fascinated by — players with 100 doubles or 100 walks. It was time consuming, but he was that sort of kid. After graduation from Yale in 1958, he used his electrical engineering degree at a series of Cold War–era firms: Raytheon, Sylvania, Mitre, and Systems Development Corporation. RAND, the quintessential postwar research outfit, had spun Systems Development Corporation off in the late 1950s as a stand-alone expert in systems analysis, which by this period entailed a substantial role for electronic computers. When Palmer started at Systems Development Corporation in 1973, his job required him to run a computing mainframe that was linked to a radar installation called Cobra Dane and located at Shemya Air Force Base off the coast of Alaska. Systems Development Corporation had been hired to support the radar computing system, the main purpose of which was to scan the skies for possible Soviet missiles. The computer was needed to calculate whether objects picked up by the radar matched the trajectory of known satellites and planes, or whether they might in fact be enemy missiles.

Palmer's training in systems and electrical engineering not only had landed him this job by the 1970s but had given him new tools to express his disappointment in the state of baseball statistics. He had continued to collect and analyze ever more expansive collections of data produced by Baseball Magazine, Who's Who in Baseball, and the Baseball Cyclopedia. By the 1960s Palmer had also been able to peruse what are widely regarded as landmark encyclopedias of baseball statistics: The Official Encyclopedia of Baseball and The Baseball Encyclopedia. The former was published by A. S. Barnes and was less a statistical repository than a biographical index, providing players' names, dates, teams, positions, games, and win/loss records (for pitchers) or batting averages (for nonpitchers). It was, in fact, a collection of glorified roster lists, but — importantly — roster lists that began in 1871. The Baseball Encyclopedia was published by Macmillan as the first extensive collection of statistical data, large enough to be known affectionately as "Big Mac."

The creation of the Macmillan guide entailed a herculean effort by a team of researchers headed by David Neft to not only gather information from a wide range of sources but also typeset the final product entirely by computer. In addition to a personal collection put together by Hall of Fame historian Lee Allen, the group drew upon the official records of the leagues. From the start of the National League in 1876, the league office would receive (or, later, designate a statistical agency to receive) a summary of statistics after every game. League rules prescribed exactly which batting, fielding, and pitching statistics were to be recorded. By 1890 the rules, while constantly in flux, specified that the statistical "summary" should contain, in addition to the basic record of hits, assists, put-outs, runs, and errors for each player, the number of earned runs by each side; the number of doubles, triples, home runs, and stolen bases made by each player; the number of double and triple plays and the players involved; the number of walks surrendered by each pitcher; the number of batsmen struck out and hit by pitch; the number of passed balls by the catcher and wild pitches by the pitcher; the time of game; and the name of the umpire.

Once the summaries arrived at the league office, they were entered into large bound ledgers, called "day-by-days" or "dailies," arranged by team and by player so that each line represented a particular performance by a player on a day. The league secretary figured out the season-ending total of doubles for a player by adding up all the doubles hit by a particular player every day throughout the season. In turn, he could figure out the total number of doubles hit by a particular team or the entire league in a season or the number of doubles hit by a player across a career.

One problem for the Macmillan researchers was that the league records only existed for the National League beginning in 1903 and the American League two years after that. So Neft and his colleagues drew from various nineteenth-century "trade papers" for baseball — Sporting Life, Sporting News, and Sporting Times — as well as over 100 local papers from 30 cities that had hosted professional baseball teams. They also turned to a collection of scrapbooks from John Tattersall. In 1941, Tattersall, then in his early 30s and working in Boston's shipbuilding industry, had the foresight to purchase a large number of baseball scrapbooks and sports pages from the Boston Transcript as it was going out of business. These dated back to 1876, when a Boston club joined the newly formed National League. Tattersall had first gained attention with his 1953 article in the Sporting News on Lajoie's 1901 batting average, having gone through the newspaper files day by day to check the totals given in the official end-of-year records. (His research adjusted Lajoie's average upward from an impressive .405 to an amazing .422.) His methods were notably low-tech — locating newspaper articles, mainly — but, when combined with other sources, they allowed researchers to delve into the earliest records of professional baseball, before the statistical summaries of games and seasons had been standardized.