BeautifulSoup (Python) and parsing HTML table -

-

##### update ###### : rendercontents() instead of contents[0] did trick. still leave open if can provide better, elegant solution!

i trying parse number of web pages desired data. table doesn't have class/id tag. have search 'website' in tr contents.

problem @ hand : displaying td.contents works fine text not hyperlinks reason? doing wrong? there better way of doing using bs in python?

those suggesting lxml, have ongoing thread here centos , lxml installation without admin privileges proving handful @ time. hence exploring beautifulsoup option.

html sample :

                   <table border="2" width="100%">                       <tbody><tr>                         <td width="33%" class="boldtd">website</td>                         <td width="33%" class="boldtd">last visited</td>                         <td width="34%" class="boldtd">last loaded</td>                       </tr>                       <tr>                         <td width="33%">                           <a href="http://google.com"></a>                         </td>                         <td width="33%">01/14/2011                                 </td>                         <td width="34%">                                 </td>                       </tr>                       <tr>                         <td width="33%">                           stackoverflow.com                         </td>                         <td width="33%">01/10/2011                                 </td>                         <td width="34%">                                 </td>                       </tr>                       <tr>                         <td width="33%">                           <a href="http://stackoverflow.com"></a>                         </td>                         <td width="33%">01/10/2011                                 </td>                         <td width="34%">                                 </td>                       </tr>                     </tbody></table>

python code far :

        f1 = open(path + "/" + file)         pagesource = f1.read()         f1.close()         soup = beautifulsoup(pagesource)         alltables = soup.findall( "table", {"border":"2", "width":"100%"} )         print "number of tables found : " , len(alltables)          table in alltables:             rows = table.findall('tr')             tr in rows:                 cols = tr.findall('td')                 td in cols:                     print td.contents[0] 

from beautifulsoup import beautifulsoup  pagesource='''...omitted brevity...'''      soup = beautifulsoup(pagesource) alltables = soup.findall( "table", {"border":"2", "width":"100%"} )  results=[] table in alltables:     rows = table.findall('tr')     lines=[]     tr in rows:         cols = tr.findall('td')         td in cols:             text=td.rendercontents().strip('\n')             lines.append(text)     text_table='\n'.join(lines)     if 'website' in text_table:         results.append(text_table)  print "number of tables found : " , len(results) result in results:     print(result)

yields

number of tables found :  1 website last visited last loaded <a href="http://google.com"></a> 01/14/2011  stackoverflow.com 01/10/2011  <a href="http://stackoverflow.com"></a> 01/10/2011

is close looking for? problem td.contents returns list of navigablestrings , soup tags. instance, running print(td.contents) might yield

['', '<a href="http://stackoverflow.com"></a>', '']

so picking off first element of list makes miss <a>-tag.


Comments

Post a Comment

Popular posts from this blog

java - SNMP4J General Variable Binding Error -

-
i trying use snmp4j snmp bulkget. when ever use snmp4j make call returned responseevent error says "general variable binding error" , data returned equal null. to debug: print out, console, exact oid , version number using in snmp4j. use printed out data snmpwalk on command line. valid results. know sending snmp4j correct pdu correct oid, version number, max repititions, etc. i have used snmp4j , code wrote succesffuly monitor other devices years. don't know different time. leaving me stumped. why "general variable binding error?" causes error? ideas debug? can reproduced? if so, first action should using wireshark or microsoft network monitor capture network packets. my guess agent gives generr response, not surprising, http://www.ietf.org/rfc/rfc1157.txt
Read more

windows - Python Service Installation - "Could not find PythonClass entry" -

-
i working on building python service. inducing error, not sure inducing it. ps c:\...> python .\file.py debug debugging service myservice - press ctrl+c stop. error 0xc00000f4 - not find service's pythonclass entry in registry error 1814 - specified resource name cannot found in image file. error 0xc0000080 - not locate module name in python class string (ie, no '.') this error coming pythonservicemessages.mc in pythonwin32. it turns out error result caused exception being thrown , python script dying off.
Read more

Determine if a XmlNode is empty or null in C#? -

-
the following code takes xmlnode data type , populates dataset object xmlnode content. write dataset's content file. public void populatedataset(xmlnode node) { xmlnodereader reader = new xmlnodereader(node); dataset ds = new dataset(); ds.readxml(reader); system.guid guid = system.guid.newguid(); string name = string.format("{0}{1}_{2}.xml", utility.xmloutputpath, utility.xmloutputfilename, guid.tostring()); //need write "node empty" file if xmlnode object empty of null ds.writexml(name, xmlwritemode.ignoreschema); } the problem encountered 1 scenario not write content file. how determine if xmlnode object null or empty? you can check if node parameter null or has innertext or innerxml properties null or empty, when enter method before creating xmlnodereader .
Read more