New Yahoo Price pages - historical downloads - healp with scraping please

[Gromley]

Hi All,

I used to find it very useful to be able to automatically scrape price history data for given stocks from the yahoo price history downloads.

The syntax used to be :

http:/ /chart.finance.yahoo.com/table.csv?s={EPIC}&a={a}&b={b}&c={c}&d={d}&e={e}&f={f}g=d&ignore=.csv

Where
{EPIC} was the stock code including the .L for uk stocks eg MKS.L and
{a}-{c} described the first date to be extracted ={a}=month, {b} = day {c}= year
{d}-{f} described the last date in the same format

However since the "upgrade" to the yahoo boards the syntax appears to have changed and is now

https:/ /query1.finance.yahoo.com/v7/finance/download/{EPIC}?period1={DATE1}&period2={DATE2}&interval=1d&events=history&crumb={CRUMBVALUE}

Where {EPIC} is as above and {DATE1} and {DATE2} are now unix datestamps.
{CRUMBVALUE} appears to be a 'randomised'(?) string which appears to be session specific, I think it is always 11 charecters.

I've worked out a semi manual way to get the right {CRUMBVALUE} (simply by looking it up before launching the scraper).

Bu my code is still failing, it generates the correct url (which when pasted into the browser produces the download csv) but trying to process this using curl & php is failing.

Before I launch into a lengthy investigation and debug - I wondered if anyone else had encountered and overcome this?

Regards,

Gromley

[kiloran]

Hi Gromley.
I don't have much time to look at this in detail at the moment, but a quick Google suggests that it's a difficult problem to overcome. This might help: https://forums.yahoo.net/t5/Yahoo-Finan ... d-p/250503 though I haven't yet looked through all 18 pages of it.

Another alternative is to use Google: https://www.google.co.uk/finance/histor ... aOUMTPmZAG
This has a download capability. I just checked an old Python script I wrote some years ago, and it is still producing data, though I haven't checked the accuracy of it. A quick glance suggests it is OK.

--kiloran

[Gromley]

Thanks Kiloran, there are so many live gripes over at Yahoo at the moment that I hadn't spotted that thread.

Google certainly would be an alternative up to point and there are others.

The thing I found particularly useful in the Yahoo data was the "Adjusted Closing Price" - where they adjust (retroactively) the price to reflect stock splits/consolidations and dividends. Not sure if any other provider offers that?

To be honest, adjustments to "regular" dividends are not so much of interest for me, but stock splits and chunky special dividends can be important to draw out.

I did look a while back at building an "events" database allowing me to make those key adjustments, so in absence of resolution of the Yahoo issue I may have to revisit that.

The bizarre thing is though that the Yahoo data in the required format is still available for manual download, they seem though just sought to make it non-scrapeable.