ajax - URL content is not accessible from HTTP client other than web browser

Question

I am trying to fetch each video segment of the m3u8 playlist (containing 1300+ segments) from this link. One's example URL is: https://s94.upstream.to/hls/o2up3xy5qaysmvk4xqksumnblvwxrpgsfhdq6igsowufgoojz5q7clj6gs4a/seg-1-v1-a1.ts. I can send successful Ajax (that returns proper content) from web browser (tested on Chrome 87 and old IE 11) - with both XHR and fetch APIs - while being on the page, from which the m3u8 playlist is originally downloaded and from the target (video segment) page, because of CORS.

However, when i try to fetch it via Node.js or Postman i receive 404 error. I set Referer, Host and Origin headers based on those in the original request i sent from browser and even set User-Agent to the Chrome's one - but i still get 404. I am aware that server can somehow check if request was sent by certain HTTP client, like web browser or Node or Postman etc. and therefore forbid/refuse to respond with a proper content. However, if i have prepared request, which is (as i consider) identical to one that i can successfully fetch data from browser, then how to check what differences i might missed that make the server refuse access to the target content on mentioned HTTP clients? How to ensure that one request is sent in identical shape on those clients?

I sometimes also get 404 when i try to reach the mentioned segment URL by copy-pasting it in the browser's address bar. Although accessing the content by clicking the request with right mouse button in Devtools Network panel always makes content to be downloaded. Why is this behavior inconsistent?

What i noticed during debugging is, when fetching is successful, then i receive Content-Type: video/MP2T header along with a CORS header, which is not the case for fetching via Node or Postman - then i receive Content-Type: text/html header (which probably indicates 404 page returned).

question from:https://stackoverflow.com/questions/65835336/url-content-is-not-accessible-from-http-client-other-than-web-browser

Answer 1

I managed to fetch the proper content from mentioned URL by using Copy as fetch option in Devtools Network panel. Apparently, some additional HTTP headers like ones prefixed with sec-fetch- were needed for server to return valid response. It worked in both Node.js and Postman.