Generating my own siteconfig

[ghost]

I am looking to generate my own siteconfig file. I can see the tags needed when I go to developer mode but the problem is just going straight to the site you won't see the epg in the source. I assume the epg is in a frame. Is there away to get to this using Tempest or is it too difficult.

https://www.samsungtvplus.com/

I know I can use the one SiteConfig but I am just trying to see if I can do it as well as it gives me more control when to grab since it only gives a few hours at a time.

[K-vanc]

Well, lets go case by case. First, still i dont check that website since its content geo-blocked, i cannot use vpn on my work pc and i am too lazy to check it on my personel pc at home but if you consider I've already shared +600 siteconfigs in this repo and have another +100 which i didnt share, answer is yes, it can be done with Tempest 😀

You can see the epg content on developer tools since it will show you the resulting html page but finding epg url may be a little tricky. You need to check all xhr, doc and even sometimes javascript request done by website during generation of epg page and find the one(s) containing epg data. Easiest way is clicking different dates or channels while developer tools is open and check newly requested urls with their content. Here, We are looking for an identifier for the channel you changed or date pattern for the date you asked in the url. Also consider, these identifiers are not always in the urls but sometimes in the request headers such as post data or cookie etc. If you can find it, next step is the mimic that request with Tempest by adding channel, date variables(if required) and/or other header parameters into related Tempest url options. Also consider that for some websites, there are additional security layers to you need to pass in order to get epg data. Thats why i have encrypted siteconfigs, in order to hide my bypass methods from users(since this is considered as hacking on some places which i dont want to teach anyone) and from websites(since if they find out, they can easily edit their code/setup to block my methods). If required, this is the real part where you have to use your skills and difficulty level changes by website. Some, i passed in seconds while some took my days or weeks to find a way..

And when finally, you get the url response, it is all about parsing data with regex. You can easily test this part with online regex websites such as regex101 which is my favorite.

Regarding url requests, i strongly suggest you to make some practice. There are several unlocked siteconfig in repo. You can see all url parameters, channel / date / subpage variables inside. Just open 1 siteconfig with your text editor and visit the same website while developer tools on. From there you can compare and understand how it is working. You may try to find and mimic same urls by yourself since there is no single way of making siteconfigs.Also you can fully debug unlocked siteconfigs to see real time url requests and resulting regex operations. When you start to understand, you can go hunt for bigger fishes.

[ghost]

Thanks for the help. Okay, I understand better now after looking at some examples and what there is already available for Samsung Plus.

The URL that contains everything was fairly easy to find

It's basically url/api/details?channel_id

I clicked on copy response in developer then pasted the response in notepad. Everything I need (epg, title, start, description) is there in json format that I could regex. Is there way for me to grab the response from the url in Tempest and then pass this to my regex?

[K-vanc]

for real testing, go into Siteconfig Maker and write a filename for your testing siteconfig. This will create a new siteconfig named as you want and save all your input in it. Then go for url1 parameter and paste the url that you believe it is for epg as below picture;

You can also add other headers if you want(and if you sure they are correct and needed since wrong header may stop server to respond). Then save this siteconfig. Now you can go Siteconfig Debugger and click to Url Debug option. This will ask Tempest to perform your request and show its result. Now, go to Launch Debugger tab and choose the siteconfig you just created as below;

When you click Debug button, it will start. If everything is ok, you will see epg data you mentioned on your screen as response of url1. From this point, your regex will join into game as Show Pattern in Siteconfig Maker. As understood from its name, Show pattern is for parsing main data into small blocks of show which each contains some information as identifier(or sometimes all data depends on source). Check below example of as how I parsed show blocks;

As you see, all show blocks splitted to have title, start time, end time, description and showicon for this example. It can be more or less. There is no mandatory structure of show blocks but in order to record them as epg, you need at least 1 start time and 1 title(for most of the cases). Once you secure and split necessary data into show blocks, all other elements such as title, desc, episode, showicon, etc.(all except channel logo), will be searched in small show blocks instead of full source. That's why making a good show block is very important and will make all other element grabbing very easy. So if I write a simple workflow;

1. Find epg url of source
2. Mimic it and get its data with Tempest Url options
3. Scrap received data into small data pieces with show pattern
4. Get all other wanted xmltv information from small data pieces with element regex patterns

This is what you need to do for most of the common/easy web sources

[ghost]

If I try to curl the url with and without the curl info from developers tool I get a 500 error. If I run it in a powershell that uses await and fetch I get a an error. If I copy response in DT I get what I need. I don't see what is being passed in the network tab to get the proper response and not a 500 error.

[K-vanc]

f I try to curl the url with and without the curl info from developers tool I get a 500 error

This means you tried url via Tempest Siteconfig and Tempest Debugger? or tried with a direct curl script/cmd request? 500 error means "internal server error". Usually happens when everything about your request is correct but there is a problem on remote server but probably they misconfigured their error codes and you are seeing this error because something is wrong on your request. Also there may be some additional security layer(s) to block direct fetching from url. Can you paste your curl or powershell request that you copied from DT?

[ghost]

I tried all of them -- shell (curl), siteconfig, debugger, and power shell.

[K-vanc]

I deleted your string since tokens in urls contains your personel information such as your ip, uuid, location etc. when decrypted properly. Anyway, I received only "ERROR_INTERNAL_ENDPOINT" message on the urls you have provided. Possible reasons are;

• you are trying to fetch from some wrong url
• server is protected against direct fetching

According to what I see on session tokens, it is using some kind of protection to block direct fetching. Also by using your valid session tokens, couldn't bypass it so probably they have been generated in another place and allowed for 1-time usage or has a very short lifespan. I don't think it will be a good challenge for a beginner 😀

[ghost]

Thanks for fixing my post. Okay, you would know better than me. I could probably do it with selenium, chromium and python assuming I can put a file instead of a url into tempest but in the end I will stick with what is on mjh.nz website.

[K-vanc]

i promise that I will have a look it on my personal pc for siteconfig creation on a day that I really don't feel lazy after work but cannot promise for the date of this 😀