[Gambas-user] httpClient does not connect to a second server

[T Lee Davidson]

On 8/30/20 1:10 PM, John Rose wrote:
> I am using the procedure shown below to connect to various servers,. 
[snip]
> 
> First time through this procedure, it works perfectly as can be seen by inspection of the HTMLandXML.txt file saved at the end: 
> I've attached the actual app's compressed source if you want to see what happens at runtime: in fact the HTMLandXML file is 
> horrible to look at and the data is better inspected at by looking at a file created later called SortedEpisodes.txt. However, 
> when the user selects a different channel (after the retrieval of data for the first selected channel) the same data is 
> retrieved and stored in the HTMLandXML.txt and SortedEpisodes.txt files.
> 
> I've put in a hClient.Close line after the data is retrieved and stored in the variable sBuffer but it didn't help. Is this a 
> bug/glitch or should I have some extra code?
> 
> Public Sub GetChannelSchedule()
> ' Download a file from the internet synchronously
> ' Put the "Async" property set to FALSE prior to Get:
> ' Get will then stop the program flow until all the information is
> ' received. In that case, you should use also the "TimeOut" property to
> ' set a timeout. If not, it could hang forever if the server does not
> ' reply properly.
>    Dim hClient As HttpClient
>    Dim sBuffer As String
>    hClient = New HttpClient As "hClient"
>    hClient.URL = "https://www.bbc.co.uk/schedules/"
>    hClient.URL &= sConnectChannelScheduleIdentifier
>    ' e.g. p00fzl6n for BBC One HD, p015pksy for BBC Two HD, p00fzl6b for BBC Four, p01kv81d for BBC Four HD
>    hClient.URL &= "/" & sYear & "/w" & sWeekNumber
>    Print "ChannelSchedule=" & hClient.URL
>    WriteLineToLogFile(hClient.URL)
>    hClient.Async = False
>    hClient.Timeout = 60
>    hClient.Encoding = "identity"
>    hClient.Get
>    If hClient.Status < 0 Then
>      QuitAfterError(sConnectMedium & " Channel " & sConnectChannel & " schedule server", "not connectable to")
>    Else
>      ' Success - read the data
>      If Lof(hClient) Then
>        sBuffer = Read #hClient, Lof(hClient)
>      Else
>        QuitAfterError(sConnectMedium & " Channel " & sConnectChannel & " schedule server", "offering no programmes")
>      Endif
>      hClient.Close
>      If Exist(sDirectory & "/HTMLandXML.txt") Then Kill sDirectory & "/HTMLandXML.txt"
>      File.Save(sDirectory & "/HTMLandXML.txt", sBuffer)
>    End If
> End ' GetChannelSchedule

Just a couple of FYI's before I get to the real issue.

You can create the URL using Replace, eg.:
hClient.URL = Replace("https://www.bbc.co.uk/schedules/&1/&2/w&3", sConnectChannelScheduleIdentifier, sYear, sWeekNumber)

I know, though, that you may wish to build the URL line by line due to your previous experience.


It is not necessary to kill "HTMLandXML.txt" prior to saving it. File.Save will overwrite it.


Now, the issue is not in GetChannelSchedule subroutine.

I ran the app three times using three different directories for file storage. On the first run, I fetched "BBC One HD". On the 
second run, I fetched "BBC Two HD". On the third run, I fetched each of those consecutively. The file size for 
PrettifiedJSON.txt on the third run is obviously the sum of its file sizes for each of the first two runs.

PrettifiedJSON.txt is created from hData in ExtractJSON(). The issue arises because hData is not being cleared prior to Add'ing 
additional collections and, therefore, the data in hData is logically corrupted.

Simply adding hData.Clear at the top of the subroutine appears to resolve the issue.

This is an example of the type of issues that can arise with the use of global variables. We sometimes forget to clear them when 
necessary. It is best practice to minimize such use. For example, the ButtonConnect_Click subroutine could be modified as follows:

Public Sub ButtonConnect_Click()
   Dim sChannelHtml As String ' No need to save the HTML text to a file
   Dim hData As Collection[] ' hData is not needed globally

   Inc Application.Busy
   SetStatusAndInstruction("Retrieving series details on " & sConnectMedium & " Channel " & sConnectChannel, "Please wait")
   sChannelHtml = GetChannelSchedule()
   hData = ExtractJSON(sChannelHtml)
   aEpisodeData = ExtractEpisodes(hData[1]["@graph"]) ' aEpisodeData assigned here to aid clarity of what is happening
   ExtractSeriesNames ' No need for a global variable for this since it merely populates a control from the global aEpisodeData
   EnableSearchControls
   DisableSelectControls
   DisableRecordControls
   ComboBoxSearchSeriesName.SetFocus
   SetStatusAndInstruction("Details of all " & sConnectMedium & " Channel " & sConnectChannel & " episodes retrieved", "Select 
Series Name")
Finally
   Application.Busy = 0
End ' ButtonConnect_Click

The applicable subroutines would of course need to be modified as appropriate. One such example is with ExtractJSON:

Public Sub ExtractJSON(sHtmlAndXml As String) As Collection[]
   Dim hHtmlDocument As New HtmlDocument
   Dim hXmlElement As XmlElement
   Dim hData As New Collection[]
   Dim sPrettyData As String
   hHtmlDocument.HtmlFromString(sHtmlAndXml)
   For Each hXmlElement In hHtmlDocument.GetElementsByTagName("script")
     If hXmlElement.Attributes["type"] <> "application/ld+json" Then Continue
     hData.Add(JSON.Decode(hXmlElement.TextContent))
   Next
   sPrettyData = PrettyFormatJSON(hData, -1)
   If Exist(sDirectory & "/PrettifiedJSON.txt") Then Kill sDirectory & "/PrettifiedJSON.txt"
   File.Save(sDirectory & "/PrettifiedJSON.txt", sPrettyData)
   Return hData
End ' ExtractJSON

There are other modifications that can be made to reduce the number of global variables. For the sake of brevity here, I've 
attached a DIFF file.


-- 
Lee
-------------- next part --------------
A non-text attachment was scrubbed...
Name: iRecorder.diff
Type: text/x-patch
Size: 2096 bytes
Desc: not available
URL: <http://lists.gambas-basic.org/pipermail/user/attachments/20200830/716f8f7d/attachment-0001.bin>