[otowi]

I am looking for a solution to archive pdf versions of papers in a database along with citiation information such as the authors, journal and abstract of the article.

I don't like EndNote, as getting the data back out is cumbersome and I don't use word for typing.

A BibTex file in a text editor like alpha is an option, and in principle you can add a link to the pdf file location on your HD. However, a database should be able to do this automatically.

Thanks for the help

otowi

[JeremyA]

I use EndNote for this (I agree - it's not very good), but I have heard of people using iPhoto to catalogue pdf files.

Jeremy

[milhouse]

You can use DevonThink to do this.

www.devon-technologies.com

It's a neat and versatile App/DB.

[biscuit]

I use EndNote and it isn't very good I agree. But it can do clever things with the formatting of references for your output. If only it wouldn't insist on putting references at the end of the Word document; I'd prefer to have it in a separate one...

I had a look at Bookends, but it didn't seem to do it for me. You might like it though, it was basically just a DB type of interface but with all the relevant fields.

Hope this helps,

biscuit

[kman42]

I've been looking for an app to do this for a LONG time. It just doesn't seem to exist.

I downloaded DevonThink, and while and interesting app, it doesn't do some of the key things that are needed. You can't seem to search the contents of a PDF, which is the most crucial feature. Another feature that would be useful is if the app renamed the files to something useful rather than 'science' or 'pdf1'.

I think what we really need is an app that can list PDFs in folders (like playlists in iTunes) and can display them in a pane on the right. In addition, it must be able to search the PDFs. It would be nice if you could add in some sort of metadata tagging (like ID3 tags) as well, but I don't know if that is supported by the PDF format. If it's not, it's a huge horrible oversight. Wouldn't it be cool if PDFs had the title, authors, abstracts, etc. in metadata format for ease of extraction?

Anyway, I'm still searching for something useful. Someone mentioned once that the full version of Adobe Acrobat could do something like what we are after, but I couldn't find anything on their website about it.

kman

[otowi]

I agree, if the database would allow searching inside the pdf's that would be great.

In the meantime I would settle for beeing able to search the tagged output you can generate with INSPEC or SciSearch and just attach the pdf/ps file of each paper to the record in the database.

I guess I have to try devon and see how far I get.

Maybe someone has used PostgreSQL or MySql for this. Not having to start from scratch is always easier.

Otowi

[milhouse]

I just dumped a few .pdfs into DevonThink. I was able to pull up the desired document when I did a search on text contained within that pdf.

I searched using "content" and "phrase" criteria and was able to retrieve the desired document 100% of the time.

I agree that the DT app should let you name documents rather than appending generic names. These can be edited, like any other saved document, after saving.

HTH and YMMV

[jessejlt]

Hmm, it would be pretty simple to write such a program, although I don't yet know how to write "Windowed" programs, because I'm a CS student and thus far we're restricted to command line apps.

SideNote: With Panther comes "Folder Actions". This would be a great folder action. A simple folder-action AppleScript that communicates to a C++ program that keeps track of the contents. Slick, and very simple.

jesse ;-)

[spiney]

I would also suggest having a look at Bookends. I left endnote for it, and I really like the attached PDF feature.

[Ookla]

There is a way to link PDFs to the EndNote reference. Open the Preferences and choose "Reference types" from the list on the left. Click the "Modify Reference Types" button and a new window will open. Scroll down to where you see the word "Image" on the left, it is the second from the bottom. In the fields to the right of that, type in whatever word you would like, I have "PDFs" under the Journal column. Click the "OK" button and save the preferences. Now, open up a reference, scroll down to the now named "PDFs" field, and choose that field. Go up to the "References" menu, choose "Insert Object" and an Open/Save dialog will appear. Navigate until you find the appropriate PDF file and click the "open" button. A small thumbnail of the PDF will now appear in the PDF field in the reference. Double click that and your PDF will open in whatever application you have set in the Finder.

It can't search the text of the PDF, DevonThink sounds like a good option. Thanks for that info. I will give DevonThink a try.

[headbirth]

Try ScrapIt Pro X...
http://www.johnvholder.com/

[kman42]

Anyone want to pitch in to write a little Cocoa app for doing this? Ideas for features?

Here's what I would like:

1) can display PDF
2) can search contents of a PDF
3) has text fields available for entering Authors, Title, Journal, etc. Possible to have these fields entered somewhat automatically after parsing the contents of the PDF
4) Multiple keyword/group assignments
5) probably should follow iTunes, iPhoto type interface

I found this for converting PDFs to text. This would be required for content searching.

kman

[Hozie]

Originally posted by kman42:
Anyone want to pitch in to write a little Cocoa app for doing this? Ideas for features?

Here's what I would like:

1) can display PDF
2) can search contents of a PDF
3) has text fields available for entering Authors, Title, Journal, etc. Possible to have these fields entered somewhat automatically after parsing the contents of the PDF
4) Multiple keyword/group assignments
5) probably should follow iTunes, iPhoto type interface

I found this for converting PDFs to text. This would be required for content searching.

kman

Sounds great... I'd actually pay for that. I would want one extra feature though: the option to rename the file according to author and/or title. It could do this by looking at the font or something and pick the text with the largest font as being the title...

[kman42]

Originally posted by Hozie:
Sounds great... I'd actually pay for that. I would want one extra feature though: the option to rename the file according to author and/or title. It could do this by looking at the font or something and pick the text with the largest font as being the title...

Yeah. I was thinking that once the PDF was converted to text, it wouldn't be too hard to at least make a guess at the title, author, etc. Make these all user-editable fields in case it screws up and give the user several options for naming the files (journal date, author, title, or any combination thereof).

kman

[Brazuca]

I've switched to BibDesk and love it. I work with TeX, so it makes sense for me. But it does give me a good way to manage my bibliography and easily export to TeX.

http://www.esm.psu.edu/mac-tex/default.html

As per the PDFs, I would also like to be able to link to a PDF file of the article, or the url, etc. I don't like the idea of a new database since it would add another degree of separation.

[m@]

BibDesk is great but a bit buggy. Hopefully version 1 will be more stable. If you use TeX it is defiantly worth a go.

[biscuit]

I have a suggestion for those of you needing a way to search PDF contents; use the Finder.

I have my 'Journals' folder set as the default place in the Find dialog. You can add a field for 'contents include' which uses the index feature of the find command. I'm not sure when the index is updated though, so if your library is quite dynamic this might not suit.

On top of this I have a folder structure as follows:

Journals/Journal Title/Year/Volume/Issue/Author, Year.pdf

This makes it easy to locate a file from a reference. I used to use 'page range.pdf' but I've found that to be less useful when people ask to see that paper by so-and-so from '98. Doing a find on the filename helps in that situation. Thinking about it, you could probably drop the Volume and Issue folders too. I also keep things in Endnote, with references linking to the nested PDFs. I also try to keep the abstracts in Endnote for easy access.

Of course, if we had a filesystem with extensible metadata and smart folders, I could have all the reference info associated with the file itself and avoid the labyrinthine folder structure. I'd still need Endnote for Word integration though.

Any thoughts? Overly organised do you think?
biscuit

[otowi]

I guess what bisquit suggests is the solution, as it requires nothing past a decent text editor like alpha. I think adding a script to generate the folder structure automatically will do the rest.

Otowi

[foamy]

I just contacted the people at Endnote (since version 7 was just announced) to see if they are planning on including this type of functionality. I will post back here if I get a response.

[kman42]

I also ran across XPDF for converting PDFs to text. It's distributed under the GNU license so we should be able to use it. I can contribute from the Cocoa/Obj-C side, but I'm not really sure how to incorporate the XPDF code. Anyone want to take a look at it?

kman

[VEGAN]

Originally posted by biscuit:
I have a suggestion for those of you needing a way to search PDF contents; use the Finder.

I have my 'Journals' folder set as the default place in the Find dialog. You can add a field for 'contents include' which uses the index feature of the find command. I'm not sure when the index is updated though, so if your library is quite dynamic this might not suit.

On top of this I have a folder structure as follows:

Journals/Journal Title/Year/Volume/Issue/Author, Year.pdf

This makes it easy to locate a file from a reference. I used to use 'page range.pdf' but I've found that to be less useful when people ask to see that paper by so-and-so from '98. Doing a find on the filename helps in that situation. Thinking about it, you could probably drop the Volume and Issue folders too. I also keep things in Endnote, with references linking to the nested PDFs. I also try to keep the abstracts in Endnote for easy access.

Of course, if we had a filesystem with extensible metadata and smart folders, I could have all the reference info associated with the file itself and avoid the labyrinthine folder structure. I'd still need Endnote for Word integration though.

Any thoughts? Overly organised do you think?
biscuit

You can also use Acrobat Reader to search a folder of PDFs for words...

[biscuit]

Originally posted by VEGAN:
You can also use Acrobat Reader to search a folder of PDFs for words...

Great, cheers for the tip! I guess this is a new feature of Adobe Reader 6? Or has it always been there?

Not nearly as quick as the Finder search though. Need to asses the quality of the results...

biscuit

[VEGAN]

Originally posted by biscuit:
Great, cheers for the tip! I guess this is a new feature of Adobe Reader 6? Or has it always been there?

Not nearly as quick as the Finder search though. Need to asses the quality of the results...

biscuit

I think it's a brand new feature of Adobe Reader 6. I just found it for myself a few weeks ago.

While slower than Finder search, it's "more precise"!

I love how it jumps to the occurrence of the searched text!!! Finder just tells you yes or no, if the word is in the file. Adobe Reader displays the page too!! Just click on the search result

[rytc]

Originally posted by otowi:

I don't like EndNote, as getting the data back out is cumbersome and I don't use word for typing.

Version 7 which apparently comes out soon is supposed to allow you to use it along with Appleworks, OpenOffice amongst others which is about time really.

[xtal]

Originally posted by rytc:
Version 7 which apparently comes out soon is supposed to allow you to use it along with Appleworks, OpenOffice amongst others which is about time really.

That is excellent news. I hope it's true. I believe that the expected release date is 'August.' We shall see.

[cowerd]

There is also this to convert .PDF to .rtf:

http://www.metaobject.com/Products.html

Scroll down to TextLightning. It also registers as a service.

[VEGAN]

Originally posted by cowerd:
There is also this to convert .PDF to .rtf:

http://www.metaobject.com/Products.html

Scroll down to TextLightning. It also registers as a service.

Yes, but I sort of a like the free `pdftotext'. It doesn't keep the fancy formating and such, but it's fast and does all I need at the moment

[voyageur]

Originally posted by otowi:

I don't like EndNote, as getting the data back out is cumbersome and I don't use word for typing.

It's very easy to get references out of EndNote with no typing at all:
1. Select references
2. Edit>Copy Formatted
3. Paste into your favorite word processor

[otowi]

Well, the last version of EndNote I used (4.1???) had a rather cumbersome way of defining custom output and input for parsing.

DevonThink is unsuitable as well for this purpose at this time:



I need to archive scientific papers which are downloaded as pdf files along with bibliographic information such as journal name, authors, abstract and so on. Is devonthink capable of parsing bibliographic as a text file that looks like this:


The upcoming Standard Edition will be able to enter structured data and will import bibliographic files (maybe not in the first release but this will definitely come).


My second question is: Can I export this information back out in the same format (brackets and identifiers) as a text file, as I will need it for citation while writing papers using bibtex? I can set up my own scripts if need be but does devonthink have a way of formatting the output of records?


Currently there are no options - im- and export "just" try to be as reversable as possible. But in version 2.0 export will be plug-in based and therefore extensible.


Best regards,


Shireen Duncan (Support)

[LeeG]

The newest version of DevonThink seems to improve on the quest many of us have. It saves an invisible text version of imported PDFs to do searching by content. Nice.

DevonThink seems to have tremendous promise. Now we NEED to be able to add searchable tags to the documents (like labels, but with user-defineable fields that data can be entered into and retrieved).

It seems there are MANY sciencetists out there using macs - and nowadays so many journals are online, there NEEDS to be a good pdf/journal article organizer - almost iPhoto like. I am shocked it doesn't exist already - the mac is made for this stuff.

Anyone feeling crafty?

Bravo Devon folks - its really coming along nicely...

Lee

[m.brown]

Originally posted by LeeG:
The newest version of DevonThink seems to improve on the quest many of us have. It saves an invisible text version of imported PDFs to do searching by content. Nice.

DevonThink seems to have tremendous promise. Now we NEED to be able to add searchable tags to the documents (like labels, but with user-defineable fields that data can be entered into and retrieved).

It seems there are MANY sciencetists out there using macs - and nowadays so many journals are online, there NEEDS to be a good pdf/journal article organizer - almost iPhoto like. I am shocked it doesn't exist already - the mac is made for this stuff.

Anyone feeling crafty?

Bravo Devon folks - its really coming along nicely...

Lee

Take a look at Books - I spoke to the author about making a "Journals" version - is anyone interested in this kind of thing - so that you could download citations from PubMed and attach the full PDF of the article?

[voyageur]

Originally posted by m.brown:
Take a look at Books - I spoke to the author about making a "Journals" version - is anyone interested in this kind of thing - so that you could download citations from PubMed and attach the full PDF of the article?

Something that did that for journal articles and was simple to learn (for busy scientists) would be ideal.

[LeeG]

YES!

If the 'books' application was altered to 'journals' and had integration with PubMed, it would rock. Of course, attachment of full text PDFs is necessary.

EndNote can actually search pubmed from within the app, and download the citation info into the reference, this kind of functionality in a PDF 'catalog'-type app would be tremendous.

Simple, easy to search with a nice aqua interface - I would buy it in a second-

Im actually shocked this doesn't exist already considering the heavy scientist usage (esp at the NIH) of macs-

Lee

[m.brown]

Originally posted by LeeG:
YES!

If the 'books' application was altered to 'journals' and had integration with PubMed, it would rock. Of course, attachment of full text PDFs is necessary.

EndNote can actually search pubmed from within the app, and download the citation info into the reference, this kind of functionality in a PDF 'catalog'-type app would be tremendous.

Simple, easy to search with a nice aqua interface - I would buy it in a second-

Im actually shocked this doesn't exist already considering the heavy scientist usage (esp at the NIH) of macs-

Lee

absolutely - it would be superb. So would anyone else be interested in usch an application?

BTW - I really don't enjoy using EndNote.

[biscuit]

Yes! This would be excellent. EndNote is not fun. Give me something simple and quick!

Then again, EndNote is handy for inserting references into manuscripts in Word and compiling a reference list at the end. I suppose this sort of functionality is a bit much to ask from shareware?

biscuit

[LeeG]

Thanks for the review, I was considering getting endnote, as I have just started a 2 year benchtop lab hiatus, but it always seemed (in prior versions) to be excessive.

I would like the following features:

-Simple logging of journal articles, searchable by defineable fields (prob the MEDLINE indexes)
-Attached PDF files (if available to the user) of full text, viewable within the app, possibly searchable (see devonthink - they convert PDF to an invisible text file that is searchable)
-Network integration with pubmed would be nice, but not needed for a 1.0 release.

I am REALLY surprised this doesn't exist, I would think this would be a standard for ANY scientist using macs (there are more than a few), considering the move to all PDF-based journal articles - say goodbye to 'lab libraries' and endless photocopying.

I am planning on using devonthink to do this for the time being (esp considering its ability to search the pdfs), but would be very interested in participating in developing a standalone streamlined app to do this.

I am not a coder, but I do have the time to extensively beta test, and real-world test to help get the software useable - anyone want to help me with the coding part?

I think OSX has built in PDF previewing, and so its really a simple database app - maybe just a filemaker-based app is what we need?

Thoughts?

Lee

[kman42]

I'm just starting to write up my thesis and am shoring up my endnote database as I get started. What a nightmare endnote is. Linking to PDFs is a pain and there is no searching.

I can't believe this app doesn't already exist. Is there a unix app that we might be able to use with x11?

kman

[Cadaver]

EndNote (vers 6 & 7), while not a great all-around app, does what it does very well - make bibliographies and reference pages.

I couldn't live without it.

[kman42]

Originally posted by Cadaver:
EndNote (vers 6 & 7), while not a great all-around app, does what it does very well - make bibliographies and reference pages.

I couldn't live without it.

Agreed. But it could and should be so much more in this day and age. Hard drives are so big and PDFs so ubiquitous in science, that it is unbelievable that ISI hasn't jumped on this possibility. It's such a huge market that they could even make it a separate app that integrates with endnote and sell it for $50. There are enough people out there who would buy it, especially if it integrated with endnote. The links to the online journal articles that are provided by many of the online services are just useless as you always have to be entering in your user id and password and it isn't very efficient for trying to find articles quickly.

kman

[riverfreak]

So everyone seems to be in agreement that Endnote is a little clunky when it comes to managing PDFs. It does however, do a very nice job at integrating searching of PubMed (with drag and drop), and of course, formatting of citations and bibliographies.

It would be a snap to build a simple database (perhaps relying on a built in MySQL or maybe filesystem db).

I've already written a little perl and mysql based generic information management system with a web-based front end. It is geared towards journal articles, but you can also add lots of other things like meeting or seminar notes, links to websites, etc. It supports uploading of files either through a file dialog or by providing a URL (in which case the file or website is fetched). It would be pretty easy to slap a cocoa front end onto this. Does someone want to collaborate on this?

edit: Please private message me if interested since I don't get around to reading all threads

[kman42]

How about this ?

I whipped it up to satisfy most of my needs. There are clearly lots of things that could be added, but it's a nice simple start.

kman

[voyageur]

Does it only work with Panther?

[kman42]

Originally posted by voyageur:
Does it only work with Panther?

Yes, sorry. I wrote it in part to explore some of the new Panther-only API calls. I am considering making it 10.2 compliant, but that will definitely have to wait until after the final 1.0 release. I will then evaluate the interest among 10.2 users.

kman

[holygoat]

I can only say BibDesk. I use DEVONthink for information management (fantastic), and it's great for searching through PDFs, but for management of BibTeX data it's BibDesk and VIM for me.

Full graphical management of BibTeX databases, searching, and you can link local files - including PDFs.
There are local_url and url fields; drag in a PDF and it sets local (then you can either open the file directly or view it in a sidebar); drag in a link and you can click it to go to the webpage.

It's perfectly stable, and does the PDF linking you want.
You may want to consider using DT and BD; BD to manage the BibTeX data, and DT to do searching and management of your thoughts.

I wouldn't mind DEVONtechnologies integrating BibTeX handling though!

[philm]

Originally posted by kman42:
How about this ?

I whipped it up to satisfy most of my needs. There are clearly lots of things that could be added, but it's a nice simple start.

kman

At first glance - looks great.

[foamy]

How about this ?

I whipped it up to satisfy most of my needs. There are clearly lots of things that could be added, but it's a nice simple start.

kman

Looks great and the pubmed search works well, but I can't get it to associate a PDF with a reference.

I put in a reference, then went to "browse for PDF", found the corresponding PDF, but when returning to Science Library, the PDF window is empty.

Edit: added a new folder to the sidebar, then added a ref and now the PDF associates fine.

Any suggestions?

Also, here's my 2c.

The pubmed search could be more like Endnote allowing for finer queries.

It would be great to have an import option for Endnote records. I.e. go into Endnote, export your library, then have Science Library parse that file and import the records directly.

Looks like a great start.

[kman42]

The app I made has been updated substantially and is now available from sourceforge. I donated it to the bibdesk project so you can go the bibdesk site on sourceforge and download the code from cvs. The app is fully contained in the 'sciencelibrary' module of the project.

If anyone wants to help out (or just try it) that is the place to go.

kman

[graveguy]

You might be interested in an app called 'Sente'...you can download it from Third Street Software. It is an iTunes like interface for managing references, complete with subsets, smart 'playlists', etc...You can attach PDFs (or any other document) to specific references and you can import/export your information to EndNote for formatting into publications...it is especially helpful for those using the PubMed database, but it can be used for any type of journal/article...

-GraveGuy

[LeeG]

WOW.

This thread keeps on going, and now we are seeing real results.

Sente is really impressive - exactly what I had in mind. I started messing around with it, and tomorrow I will try to corral my recent lit searches into its format - if all goes well, I have a new favorite program.

The 'continuously updated' searches are a fantastic feature, esp if you focus on a few recurring topics, and it links nicely to local PDFs. Fantastic.

A bit pricey, considering what iLife gives you for $49, but I may have to splurge as the program looks great-

Lee

[kman42]

Originally posted by LeeG:
WOW.

This thread keeps on going, and now we are seeing real results.

Sente is really impressive - exactly what I had in mind. I started messing around with it, and tomorrow I will try to corral my recent lit searches into its format - if all goes well, I have a new favorite program.

The 'continuously updated' searches are a fantastic feature, esp if you focus on a few recurring topics, and it links nicely to local PDFs. Fantastic.

A bit pricey, considering what iLife gives you for $49, but I may have to splurge as the program looks great-

Lee

You might check out my app that I mentioned above. It has a lot of the same features a Sente, but it is not quite as sophisticated. You can create groups of references, set up recurrent searches, link local PDFs (it even renames them and consolidates them into a reasonable directory structure). Best of all it is free and open source. It's available on sourceforge on the bibdesk site through cvs (you'll have to compile it, but that's not too bad).

kman