CrossRef Labs is happy to announce the first public release of "pdf-extract" an open source set of tools and libraries for extracting citation references (and, eventually, other semantic metadata) from PDFs. We first demonstrated this tool to CrossRef members at our annual meeting last year. See the pdf-extract labs page for a detailed introduction to this new set of tools.

If you are unable to download and install the tool, you can play with a experimental web interface called "Extracto." Be warned, Extracto is running on very feeble server using an erratic and slow internet connection. The only guarantee that we can make about using it is that it will repeatedly fall over and annoy you. The weasel has spoken.
]]> http://www.crossref.org/CrossTech/2012/04/pdf-extract.html http://www.crossref.org/CrossTech/2012/04/pdf-extract.html Citation Formats Tue, 17 Apr 2012 06:56:58 -0500

PHD Comics has posted its Valentine's Day Reading list.

Without DOIs! 

So in order to preserve the scholarly citation record, we've resolved those that have DOIs....

Title:  The St. Valentine's Day Frontal Passage

Citation:  Sassen, K, 1980, 'The St. Valentine's Day Frontal Passage', Bulletin of the American Meteorological Society, vol. 61, no. 2, p. 122.

CrossRef DOI:  http://dx.doi.org/10.1175/1520-0477(1980)061<0122:TSVDFP>2.0.CO;2

Title:  SUICIDE AND HOMICIDE ON ST. VALENTINE'S DAY

Citation:  LESTER, D, 1990, 'SUICIDE AND HOMICIDE ON ST. VALENTINE'S DAY', Perceptual and Motor Skills, vol. 71, no. 7, p. 994.

CrossRef DOI:  http://dx.doi.org/10.2466/PMS.71.7.994-994

Title:  The St. Valentineʼs Day Massacre

Citation:  Eckert, W, 1980, 'The St. Valentineʼs Day Massacre', The American Journal of Forensic Medicine and Pathology, vol. 1, no. 1, pp. 67-70.

CrossRef DOI:  http://dx.doi.org/10.1097/00000433-198003000-00011

Title:  For Valentine's Day

Citation:  Kutzner, H, 2001, 'For Valentine's Day', Cancer, vol. 91, no. 4, pp. 804-805.

CrossRef DOI:  http://dx.doi.org/10.1002/1097-0142(20010215)91:4<804::AID-CNCR1067>3.3.CO;2-K

First off, if you'd like to try citation formatting without using content negotiation, there's a simple web UI that allows input of a DOI, style and locale selection.

If you're more into accessing the web via your favorite programming language, have a look at these content negotiation curl examples. To make a request for the new "text/bibliography" content type:

$ curl -LH "Accept: text/bibliography; style=bibtex" http://dx.doi.org/10.1038/nrd842

@article{Atkins_Gershell_2002, title={From the analyst's couch: Selective anticancer drugs}, volume={1}, DOI={10.1038/nrd842}, number={7}, journal={Nature Reviews Drug Discovery}, author={Atkins, Joshua H. and Gershell, Leland J.}, year={2002}, month={Jul}, pages={491-492}}

A locale can be specified with the "locale" content type parameter, like this:

$ curl -LH "Accept: text/bibliography; style=mla; locale=fr-FR" http://dx.doi.org/10.1038/nrd842

Atkins, Joshua H., et Leland J. Gershell. « From the analyst's couch: Selective anticancer drugs ». Nature Reviews Drug Discovery 1.7 (2002): 491-492.

You may want to process metadata through CSL yourself. For this use case, there's another new content type, "application/citeproc+json" that returns metadata in a citeproc-friendly JSON form:

$ curl -LH "Accept: application/citeproc+json" http://dx.doi.org/10.1038/nrd842

{"volume":"1","issue":"7","DOI":"10.1038/nrd842","title":"From the analyst's couch: Selective anticancer drugs","container-title":"Nature Reviews Drug Discovery","issued":{"date-parts":[[2002,7]]},"author":[{"family":"Atkins","given":"Joshua H."},{"family":"Gershell","given":"Leland J."}],"page":"491-492","type":"article-journal"}

Finally, to retrieve lists of supported styles and locales, either hit these URLs:

• http://data.crossref.org/styles
• http://data.crossref.org/locales

or check out the CSL style and locale repositories.

There's one big caveat to all this. The CSL processor will do its best with CrossRef metadata which can unfortunately be quite patchy at times. There may be pieces of metadata missing, inaccurate metadata or even metadata items stored under the wrong field, all resulting in odd-looking formatted citations. Most of the time, though, it works.

The service provides an API that will attempt to reverse look-up a URL to a DOI and return the membership status (member or non-member) of the root domain of the URL. In practice resolving URLs to DOIs has substantial limitations - many publishers redirect the resolution URL of DOIs to other online content and URLs become clogged up with session IDs and other cruft appearing in their query parameters. All of this means it is unlikely that the URLs that appear to be the end result of DOI resolution are actually the URLs pointed to.

However, it's also possible to provide only a host name, in which case, as with a URL, the CrossRef membership status for the root domain will be returned.

There's also a downloadable list of hashed domains that belong to CrossRef members which will be useful to those who want to determine the membership status of a domain locally. Also, a bookmarklet allows anyone to easily check a web page they are looking at to see if the domain it is hosted on belongs to a CrossRef member.

Check it out over at the documentation page.

Edward Zukowski (DataCite's resident propellor-head) took up the challenge with gusto and, as of September 22nd DataCite has also been supporting content negotiation for its DOIs. This means that one million more DOIs are now linked-data friendly. Congratulations to Ed and the rest of the team at DataCite.

We hope this is a trend. Back in June Knowledge Exchange organized a seminar on Persistent Object Identifiers. One of the outcomes of the meeting was "Den Haag Manifesto" a document outlining five relatively simple steps that different persistent identifier systems could take in order to increase interoperability. Most of these steps involved adopting linked data principles including support for content negotiation. We look forward to hearing about other persistent identifiers adopting these principles over the next year.

Having said that, this time I will refrain from calling-out anybody specifically...

The web API comes with two services: a family name detector that will pick out potential family names from chunks of text and a family name autocompletion system.

Very brief documentation can be found here along with a jQuery example of autocompletion.

The database is still in development so there may be some oddities and inaccuracies in there. Right now one obvious omission from the name list that I hope to address soon are double-worded names such as "von Neumann". We're not proposing this database as an authority but rather something that backs a practical service for family name detection and autocompletion.

Well, we went with option "D."

From now on, DOIs, expressed as HTTP URIs, can be used with content-negotiation.

Let's get straight to the point. If you have curl installed, you can start playing with content-negotiation and CrossRef DOIs right away:

curl -D - -L -H   "Accept: application/rdf+xml" "http://dx.doi.org/10.1126/science.1157784" 

curl -D - -L -H   "Accept: text/turtle" "http://dx.doi.org/10.1126/science.1157784"

curl -D - -L -H   "Accept: application/atom+xml" "http://dx.doi.org/10.1126/science.1157784"

Or if you are already using CrossRef's "unixref" format:

curl -D - -L -H "Accept: application/unixref+xml" "http://dx.doi.org/10.1126/science.1157784" 

This will work with over 46 million CrossRef DOIs as of today, but the beauty of the setup is that from now on, any DOI registration agency can enable content negotiation for their constituencies as well. DataCite- we're looking at you ;-) .

It also means that, as registration agency members (CrossRef publishers, for instance) start providing more complete and richer representations of their content, we can simply redirect content-negotiated requests directly to them.

We expect that that this development will round-out CrossRef's efforts to support standard APIs including OpenURL and OAI_PMH and we look forward to seeing DOIs increasingly used in linked data applications.

Finally, CrossRef would just like to thank the IDF and CNRI for their hard work on this as well as Tony Hammond and Leigh Dodds for their valuable advice and persistent goading.

The TWG Discussion forum replaces the TWG mailing list and can be accessed by members of the CrossRef community who log in to our support portal. Intended topics include technical matters related to Crossref's services, DOI issues and Crossref system operation.

Some more PDF-related tools will follow soon.

• InChIs: the bigger the molecule the longer the InChI, so we've standardized on the fixed-length InChIKey. This doesn't mean anything on its own, so we've gone the Semantic Web route of including an InChI resolver HTTP URI. Alternatively you can extract the InChIKeys with a regular expression.
• Ontology terms: we're using HTTP URIs again and pointing to either Open Biomedical Ontology URIs (biology, biomedicine; slashy) or RSC ontology terms (chemistry; hashy). Often the OBO URIs resolve to a specific web page, but for the moment the RSC URIs just point to a large OWL file. Slashy URIs are quite a bit more involved so we'll have to see what the demand is like.

There's only about 4K to play with, so it's only ever going to be a best-of. More detailed article metadata has to go in either a sidecar file, as Tony has pointed out before, or ideally on the article landing page. The example files are here and I've posted something with a different slant on the RSC technical blog.

doi:10.1045/july2010-hammond

I thought I could take this opportunity to demonstrate one evolution path from traditional record-based search to a more contemporary triple-based search. The aim is to show that these two modes of search do not have to be alternative approaches but can co-exist within a single workflow.

Let me first mention a couple of terms I’m using here: ‘graphs’ and ‘properties’. I’m using ‘property’ loosely to refer to the individual RDF statement (or triple) containing a property, i.e. a triple is a ‘(subject, property, value)’ assertion. And a ‘graph’ is just a collection of ‘properties’ (or, more properly, triples). Oh, and I’ll also use the term ‘records’ when considering ‘graphs’ as pre-fabricated objects returned within a result set.

So, what do we have here? We have on the left a traditional means of disseminating search results which is typically record based. A new set of records may be generated by querying using the API provided, whether proprietary or public such as Lucene or SRU/CQL. We can thus consider this search service as a ‘record store’ – even though records tend to generated anew rather than retrieved. The individual records in the result set are collections or groupings of ‘properties’ about the subjects of the query. Note that this is somewhat similar to the way music is packaged for physical distribution with many tracks (‘properties’) combined onto a single album (‘record’ or ‘graph’) which contains a thematic coherence – either same artist or compilation around a given topic.

Digital music distribution, on the other hand, allows for albums to be atomized so that individual tracks may be cherry-picked at will. This is not dissimilar from what happens in a ‘triple store’ where the basic properties (‘tracks’) that in a regular search engine were together combined in a ‘record’ (‘album’) to present a search result can now be plucked apart and recombined into newer bespoke ensembles. Note that this querying and recombination can be applied across the full triple store or even across this triple store and remote triple stores since the same data model is applied. Certainly, at the data model level federated searching thus becomes a non-issue.

Suppose now that our search server (or record store) is an OpenSearch-type service, i.e. the result sets are distributed as some list-based format, typically RSS, and that the list-based format either provides an RDF graph or can be transformed to such a graph, we could then use that as a basis for feeding an RDF triple store.

So, now then at right we have a triple store which is a large database of triples (or properties) compiled from all the records in the record store. And since this is a triple store we can query it using SPARQL. For example, this trival SPARQL query:

PREFIX dc: <http://purl.org/dc/elements/1.1/>
PREFIX prism: <http://prismstandard.org/namespaces/basic/2.0/>
SELECT ?doi ?title
  WHERE {
    ?s prism:doi ?doi .
    ?s dc:title ?title .
    FILTER regex(?title, "boson", "i" ) 
  }
LIMIT 5

returns the first five articles (referenced by DOI) with title containing the word ‘boson’:

--------------------------------------------------------------------------------------------------
| doi                   | title                                                                  |
==================================================================================================
| "10.1038/nature05513" | "Comparison of the Hanbury Brown–Twiss effect for bosons and fermions" |
| "10.1038/221999a0"    | "Physics: The Intermediate Boson"                                      |
| "10.1038/313506b0"    | "The nuts and bolts of bosons"                                         |
| "10.1038/301287a0"    | "The search for bosons: A golden year for the weak force"              |
| "10.1038/424003a"     | "Below-par performance hampers Fermilab quest for Higgs boson"         |
--------------------------------------------------------------------------------------------------

Now let’s contrast this with a conventional record-based search, such as shown at left, to find the first five articles (referenced by DOI) with title containing the word ‘boson’ would use a query (here SRU/CQL, and CQL is bolded) such as:

?query=dc.title="boson"&maximumRecords=5&httpAccept=application/rss+xml

and would receive a set of result records (here RSS) like so:

...
<item rdf:about="http://dx.doi.org/10.1038/nature05513">
  <title>Comparison of the Hanbury Brown–Twiss effect for bosons and fermions</title>
  <link>http://dx.doi.org/10.1038/nature05513</link>
  <dc:identifier>doi:10.1038/nature05513</dc:identifier>
  <dc:title>Comparison of the Hanbury Brown–Twiss effect for bosons and fermions</dc:title>
  ...
</item>
<item rdf:about="http://dx.doi.org/10.1038/221999a0">
  <title>Physics: The Intermediate Boson</title>
  <link>http://dx.doi.org/10.1038/221999a0</link>
  <dc:identifier>doi:10.1038/221999a0</dc:identifier>
  <dc:title>Physics: The Intermediate Boson</dc:title>
  ...
</item>
...

Note also that there is an interesting halfway house as shown in the diagram, whereby a set of result records presenting a single RDF graph can be queried as its own (very) restricted triple store.

In general, because a triple store is so primitive and it can be queried alongside other triple stores the queries that can be put together can be highly complex and customized with arbitrary data. The result from such a query differs from a traditional ‘record’ where a fixed property set is bound together in a presentation. Such a result is user-determined as opposed to the server-determined nature of traditional result ‘records’.

I hope that this post has been able to show in some degree that although there are some obvious differences there is nevertheless a synergy between these two modes of searching: prêt-à-porter and tailored.

I think that several of us were struck by how little actually needs to be done in order to fully address virtually all of the concerns that the linked data community has expressed about DOIs. Not only that- but in some of these scenarios we would put ourselves in a position to be able to semantically-enable over 40 million DOIs with what amounts to the flick of a switch.

Given the huge interest in linked data on the part of researchers and CrossRef members- it seems like it would be a fantastic boon to both the IDF (International DOI Foundation) and CrossRef if we were able to do something quickly here.

Anyway- The following are notes outlining several concrete proposals for addressing the limitations of DOIs as identifiers in linked data applications. They range in complexity/effort involved- with the simplest scenario providing minimal (yet functional) LD capabilities for just one RA's members (CrossRef's) and the most complex providing per-RA and per-RA-member configurability on how DOIs would behave for LD applications.

We'd appreciate comments, questions, suggestions, corrections, etc.

A: Simplest Scenario

What would need to be done?

1. CrossRef implements a linked data service. For example, hosted at rdf.crossref.org.
2. CrossRef recommends that any member publisher who wants to add rudimentary linked data capabilities to their site could simply insert some simple link elements into their landing Pages. So, for instance, for the article with the DOI 10.5555/1234567 in the Journal of Psychoceramics, the publisher would put the following in the landing page for the article:

How it would work

1. A sem-web-enabled browser would query dx.doi.org/10.5555/1234567 and get a normal 302 redirect to the publisher's landing page. 
2. The sem-web-enabled browser would sniff the page for the link elements and retrieve the representations it wanted from rdf.crossref.org
3. The returned document would contain an appropriate representation of the metadata that the publisher has deposited with CrossRef. It would also assert that:

Pros

1. Doesn't require changes at DOI/Handle levels
2. Is easy for publisher to opt-in or opt-out
3. Requires minimal development on the part of CrossRef.

Cons

1. Only applies to CrossRef DOIs.
2. It depends on publishers taking action. Might be a long time before publishers add the needed links to their landing pages or support content negotiation.
3. DOI system is still not strictly LD compliant (e.g. it is returning 302 redirects. Naive sem-web browsers might 'stop' after getting a 302. Should ideally use 303s, content negotiation, etc.)
4. Doesn't work for DOIs that currently bypass landing pages and which go directly to content.

B: Simple + IDF Global Semantic Compliance

What would need to be done?

1. Same as "Simplest Scenario"
2. IDF globally changes dx.doi.org to return 303 redirect

How would it work?

Pros

1. All DOIs conform to expectations for LD identifiers
2. Easy for publisher to opt-in or opt-out
3. Requires minimal development on part of CrossRef
4. Requires minimal work (?) on part of IDF

Cons

1. Requires global change on part of IDF. Global change might conflict with requirements of other RAs.
2. It depends on publishers taking action. Might be a long time before publishers add needed links to their landing pages or support content negotiation.
3. Doesn't work for DOIs that currently bypass landing pages (e.g. OECD spreadhseets, UICR datasets, etc.)

C: Simple + IDF Global Semantic Compliance + RA CN Intercept

What would need to be done?

1. Same as "B: Simple + IDF Global Semantic Compliance" Scenario
2. IDF  changes dx.doi.org to redirect content-negotiated dx.doi.org queries to RA-controlled resolver depending on the preferences of the RA.
3. RA implements DOI resolver (e.g. dx.crossref.org) that supports content negotiation. RA allows its members to specify to the RA  that they want either:
1. RA to forward all requests to the member's site.
2. RA to "intercept" content-negotiations for non-HTML representations and direct them appropriately (e.g. return appropriate representation from rdf.crossref.org)

How would it work?

Pros

1. All DOIs conform to expectations for LD identifiers
2. Allows RA to potentially LD-enable its members very quickly.
3. Easy for ra-members to opt-in or opt-out
4. Requires minimal development on part of CrossRef
5. Would even work for DOIs that bypass landing pages

Cons

1. Requires global change on part of IDF. Global change might conflict with requirements of other RAs.
2. Requires change to add decision logic implementation on part of IDF. 
3. Requires development of RA resolvers that implement per-member resolution logic (note- this would probably actually be done at DOI level)

D: Simple + IDF Selective Semantic Compliance + RA CN Intercept

What would need to be done?

1. Same as Simplest Scenario
2. IDF  changes dx.doi.org to return either 302 or 303 redirect depending on the preferences of the RA.
3. IDF  changes dx.doi.org to redirect content-negotiated dx.doi.org queries to RA-controlled resolver depending on the preferences of the RA.
4. RA implements DOI resolver (e.g. dx.crossref.org) that supports content negotiation. RA allows its members to specify to the RA  that they want either:
1. RA to forward all requests to the member's site.
2. RA to "intercept" content-negotiations for non-HTML representations and direct them appropriately (e.g. return appropriate representation from rdf.crossref.org)

How would it work?

Pros

1. Allows RA to potentially LD-enable its members very quickly.
2. Easy for ra-members to opt-in or opt-out
3. Requires minimal development on part of CrossRef
4. Would even work for DOIs that bypass landing pages

Cons

1. Only some DOIs conform to expectations for LD identifiers
2. Requires change to add decision logic implementation on part of IDF. 
3. Requires development of RA resolvers that implement per-member resolution logic (note- this would probably actually be done at DOI level)

Hi Geoff:

First off, there is no question but that CrossRef was established to take on the reference linking challenge for scholarly literature. (Hell, it's there, as you point out, in the organization name - PILA - as well as in the application name - CrossRef.)

But one should also remember that DOI as it was sold at the time was promising so much more. I disagree with you that the participants back then were as wholly innocent of the FRBR terms as you might suggest. Certainly there were ample presentations on DOI that sought to elucidate those relationships.

No matter. FRBR is a useful reference model to clarify some of these concepts. But not one that we are overly concerned with at this time. Nor even whether DOI maps one to one onto a given FRBR layer. What we are more concerned with on a pragmatic level is how DOI maps onto the Web architecture and especially how it plays along with Linked Data concepts.

(Aside: A propos FRBR we might be in danger of repeating the OSI mistake for standardizing the network layer model. Ultimately that was maintained as a reference model but dropped as a concrete model in favour of the TCP/IP stack. Could be that FRBR is our OSI and Linked Data is our TCP/IP stack? That is, we might have to settle on the coarser data model in order to get a coherent story out the door where all can agree.)

You say:

"we need a mechanism to distinguish between when we are getting the thing pointed to by the CrossRef DOI (the PDF , HTML, etc.) as opposed to "something about the thing" (e.g. the landing page, metadata record, etc.)"

Now is this something that CrossRef can truly distinguish and make apparent in its service architecture? If we retain the notion of landing page we are already essentially saying that a CrossRef HTTP URI identifies a decsription of the resource, i.e. a Non-Information Resource, or Other Resource, and that is properly indicated within the architecture by returning a "303 See Other status" code.

I think that's all we're saying at the moment as a first step.

Web architecture wants to know if the DOI HTTP URI is a thing or description of a thing. I say the latter. You seem to suggest in your comment the latter too. I wonder if we could get a vote on that.

And btw, I am not suggesting that CrossRef needs to dive into the business of "tracking compoend documents in their entirety". Far from it. Lets just get a common resource architecture agreed publicly and then we can build on that.

This observation I received in a private email is something I fully support:

"The real problem is what doi http uri identify on the web. Everything flows from the answer to that Q."

When the CrossRef linking system was developed it was focused primarily on facilitating persistent linking amongst journals and conference proceedings. The system was quickly adapted to handle books and more recently to handle working papers, technical reports, standards and “components”- a catchall term used to refer to everything from individual article images to database records.

In practice the content outside of the core journals and conference proceedings has accounted for relatively low volume. However, we expect that over the next few years this will change and that books and databases will increasingly drive the future growth in CrossRef’s citation linking services. Interestingly, these content types all share characteristics that make them substantially different from the journals and conference proceedings that we have hitherto focused on.

Both books and databases introduce new challenges to technology and policies of our citation linking service. The challenges revolved around two areas:

• Structure: Both books and databases can have complex structures and the publishers of this content are likely to require granular identification of these content substructures along with a mechanism for documenting the relationship between these substructures (e.g. this section is part of this chapter which is part of this monograph which is part of this series)
• Versioning: Unlike typical journals and conference proceedings, books and database records sometimes change over time.

When confronted with the issues of structure and versioning publishers are often tempted to take shortcuts and decide to simply assign DOIs at the highest level structure and to the “work” instead of a particular “manifestation” or version of that work. Indeed, section 5.5 of CrossRef's DOI Name Information and Guidelines recommends this. But this approach could have a negative impact on the integrity of the scholarly citation record that CrossRef is attempting to maintain.

Fundamentally, CrossRef DOIs are aimed at providing a persistent online citation infrastructure for scholarly and professional publishers. Consequently, decisions about where to apply CrossRef DOIs should be guided by common expectations about the way in which citations work. Citations are typically used to credit ideas or provide evidence. A reader follows a citation in order to obtain more detail or to verify that an author is accurately representing the item cited. A rule of thumb is that a reader has a reasonable expectation that when they follow a citation, they will be taken to what the author saw when creating the citation. Any divergent behavior could result in the reader concluding that the author was misrepresenting the item cited. A further implication of this is that any changes to content that are likely to effect the crediting or interpretation of the content should result in that changed content getting a new CrossRef DOI.

Typically, this means that CrossRef DOIs should be probably assigned at the expression level and different expressions should be assigned different CrossRef DOIs. This is because assigning a CrossRef DOI at the higher "work" level is generally not granular enough to guarantee that a reader following the citation will see what the author saw when creating the citation. For example, one translation of a work might be substantially different from another translation of the same work. Similarly a draft version of a work might be substantially different from the final published version of the work. In each case, resolving a citation to a different expression of the work than the expression that was originally cited might result in the reader interpreting the content differently than the citing author.

In general, different "equivalent manifestations" of the same work can safely be assigned the same CrossRef DOI. So, for instance, the HTML formatted version an article and the PDF formatted version of an article can almost always be assigned the same CrossRef DOI. Any differences between the two are unlikely to affect the crediting of, or reader's interpretation of, the work. But sometimes it is even possible that different manifestations of an expression will differ enough to merit different CrossRef DOIs. For instance, a semantically enhanced version of an article might require new crediting (e.g. the parties responsible for adding the semantic information) and the resulting semantic enhancement may conceivably alter the reader's interpretation of the article.

Unfortunately, there is no hard and fast rule about where and when to assign new CrossRef DOIs. Instead there is only a guideline, namely:

"Assign new CrossRef DOIs to content in a way that will ensure that a reader following the citation will see something as close to what the original author cited as is possible."

The implications of this to publishers are important, especially when they are assigning DOIs to protean content types. For instance, it may mean that:

• Book publishers should be expected to keep old editions of books available for link resolution purposes.
• Publishers of content that can change rapidly (e.g. by the second) should provide facilities for creating frozen, archived snapshots of content for citation purposes.
• All publishers of protean content should issue guidelines instructing researchers on when it is appropriate to cite a work, manifestation or version.

CrossRef needs to actively consider these issues as publishers start assigning CrossRef DOIs to more dynamic types of content. Minimally, we should be able to provide publishers with recommendations on how to make dynamic content citable. We may even want to consider enshrining certain types of behavior in our terms and conditions so as to ensure the future integrity of the scholarly citation record.

In short, we need to update our guidelines.