[Gambas-user] httpClient does not connect to a second server

[John Rose]

On 30/08/2020 23:26, user-request at lists.gambas-basic.org wrote:
> On 8/30/20 1:10 PM, John Rose wrote:
>> I am using the procedure shown below to connect to various servers,. 
> [snip]
>>
>> First time through this procedure, it works perfectly as can be seen 
>> by inspection of the HTMLandXML.txt file saved at the end: I've 
>> attached the actual app's compressed source if you want to see what 
>> happens at runtime: in fact the HTMLandXML file is horrible to look 
>> at and the data is better inspected at by looking at a file created 
>> later called SortedEpisodes.txt. However, when the user selects a 
>> different channel (after the retrieval of data for the first selected 
>> channel) the same data is retrieved and stored in the HTMLandXML.txt 
>> and SortedEpisodes.txt files.
>>
>> I've put in a hClient.Close line after the data is retrieved and 
>> stored in the variable sBuffer but it didn't help. Is this a 
>> bug/glitch or should I have some extra code?
>>
>> Public Sub GetChannelSchedule()
>> ' Download a file from the internet synchronously
>> ' Put the "Async" property set to FALSE prior to Get:
>> ' Get will then stop the program flow until all the information is
>> ' received. In that case, you should use also the "TimeOut" property to
>> ' set a timeout. If not, it could hang forever if the server does not
>> ' reply properly.
>>    Dim hClient As HttpClient
>>    Dim sBuffer As String
>>    hClient = New HttpClient As "hClient"
>>    hClient.URL = "https://www.bbc.co.uk/schedules/"
>>    hClient.URL &= sConnectChannelScheduleIdentifier
>>    ' e.g. p00fzl6n for BBC One HD, p015pksy for BBC Two HD, p00fzl6b 
>> for BBC Four, p01kv81d for BBC Four HD
>>    hClient.URL &= "/" & sYear & "/w" & sWeekNumber
>>    Print "ChannelSchedule=" & hClient.URL
>>    WriteLineToLogFile(hClient.URL)
>>    hClient.Async = False
>>    hClient.Timeout = 60
>>    hClient.Encoding = "identity"
>>    hClient.Get
>>    If hClient.Status < 0 Then
>>      QuitAfterError(sConnectMedium & " Channel " & sConnectChannel & 
>> " schedule server", "not connectable to")
>>    Else
>>      ' Success - read the data
>>      If Lof(hClient) Then
>>        sBuffer = Read #hClient, Lof(hClient)
>>      Else
>>        QuitAfterError(sConnectMedium & " Channel " & sConnectChannel 
>> & " schedule server", "offering no programmes")
>>      Endif
>>      hClient.Close
>>      If Exist(sDirectory & "/HTMLandXML.txt") Then Kill sDirectory & 
>> "/HTMLandXML.txt"
>>      File.Save(sDirectory & "/HTMLandXML.txt", sBuffer)
>>    End If
>> End ' GetChannelSchedule
>
> Just a couple of FYI's before I get to the real issue.
>
> You can create the URL using Replace, eg.:
> hClient.URL = Replace("https://www.bbc.co.uk/schedules/&1/&2/w&3", 
> sConnectChannelScheduleIdentifier, sYear, sWeekNumber)
>
> I know, though, that you may wish to build the URL line by line due to 
> your previous experience.
>
>
> It is not necessary to kill "HTMLandXML.txt" prior to saving it. 
> File.Save will overwrite it.
>
>
> Now, the issue is not in GetChannelSchedule subroutine.
>
> I ran the app three times using three different directories for file 
> storage. On the first run, I fetched "BBC One HD". On the second run, 
> I fetched "BBC Two HD". On the third run, I fetched each of those 
> consecutively. The file size for PrettifiedJSON.txt on the third run 
> is obviously the sum of its file sizes for each of the first two runs.
>
> PrettifiedJSON.txt is created from hData in ExtractJSON(). The issue 
> arises because hData is not being cleared prior to Add'ing additional 
> collections and, therefore, the data in hData is logically corrupted.
>
> Simply adding hData.Clear at the top of the subroutine appears to 
> resolve the issue.
>
> This is an example of the type of issues that can arise with the use 
> of global variables. We sometimes forget to clear them when necessary. 
> It is best practice to minimize such use. For example, the 
> ButtonConnect_Click subroutine could be modified as follows:
>
> Public Sub ButtonConnect_Click()
>   Dim sChannelHtml As String ' No need to save the HTML text to a file
>   Dim hData As Collection[] ' hData is not needed globally
>
>   Inc Application.Busy
>   SetStatusAndInstruction("Retrieving series details on " & 
> sConnectMedium & " Channel " & sConnectChannel, "Please wait")
>   sChannelHtml = GetChannelSchedule()
>   hData = ExtractJSON(sChannelHtml)
>   aEpisodeData = ExtractEpisodes(hData[1]["@graph"]) ' aEpisodeData 
> assigned here to aid clarity of what is happening
>   ExtractSeriesNames ' No need for a global variable for this since it 
> merely populates a control from the global aEpisodeData
>   EnableSearchControls
>   DisableSelectControls
>   DisableRecordControls
>   ComboBoxSearchSeriesName.SetFocus
>   SetStatusAndInstruction("Details of all " & sConnectMedium & " 
> Channel " & sConnectChannel & " episodes retrieved", "Select Series 
> Name")
> Finally
>   Application.Busy = 0
> End ' ButtonConnect_Click
>
> The applicable subroutines would of course need to be modified as 
> appropriate. One such example is with ExtractJSON:
>
> Public Sub ExtractJSON(sHtmlAndXml As String) As Collection[]
>   Dim hHtmlDocument As New HtmlDocument
>   Dim hXmlElement As XmlElement
>   Dim hData As New Collection[]
>   Dim sPrettyData As String
>   hHtmlDocument.HtmlFromString(sHtmlAndXml)
>   For Each hXmlElement In hHtmlDocument.GetElementsByTagName("script")
>     If hXmlElement.Attributes["type"] <> "application/ld+json" Then 
> Continue
>     hData.Add(JSON.Decode(hXmlElement.TextContent))
>   Next
>   sPrettyData = PrettyFormatJSON(hData, -1)
>   If Exist(sDirectory & "/PrettifiedJSON.txt") Then Kill sDirectory & 
> "/PrettifiedJSON.txt"
>   File.Save(sDirectory & "/PrettifiedJSON.txt", sPrettyData)
>   Return hData
> End ' ExtractJSON
>
> There are other modifications that can be made to reduce the number of 
> global variables. For the sake of brevity here, I've attached a DIFF file.

Thanks Lee for the above. I've implemented the modifications from your 
diff file. Only problem is "Global variable hidden by local declaration 
: aEpisodeData" in ExtractEpisodes procedure. I guess that won't make 
any difference at runtime but it would be nice not to have it. I thought 
of using the idea (as per hData & hdataLocal in your diff file) of 
changing the Dim name from aEpisodeData to aEpisodeDataLocal in the 
procedure (together with its uses in the procedure) but that then causes 
changes to the definition & use in the SortEpisodeData procedure. Any 
better ideas to prevent the compilation message?

-- 
John

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gambas-basic.org/pipermail/user/attachments/20200831/6a12c99e/attachment-0001.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: iRecorder.tar.gz
Type: application/gzip
Size: 34287 bytes
Desc: not available
URL: <http://lists.gambas-basic.org/pipermail/user/attachments/20200831/6a12c99e/attachment-0001.gz>