Adding Digital Cable Channels (For ATSC/QAM Tuner Cards -- USA/Canada)

From MythTV Official Wiki
Revision as of 16:56, 20 June 2010 by Wagnerrp (Talk | contribs)

Jump to: navigation, search

Important.png Note: For those Comcast customers in an area where the Pace DC50X, Motorola or Thomson DTA's are being deployed, Comcast is broadcasting virtual channel maps that show QAM channel layouts. scte65scan is a utility that will read these maps.

For more details, see Scte65scan


Important.png Note: MythTV as of 0.20 has an easier to use method, however the below info may still be useful if the builtin channel scanner fails to find your channels.

Instead of hacking the database you should scan for channels in mythtv-setup and then use the frontend channel editor, 'E' in LiveTV mode, to edit the channels.

When scanning for channels you probably want to select the QAM-256 modulation and the standard cable frequencies, but it is possible your cable operator uses HRC or IRC. Some rural operators use QAM-64. It is possible to scan only the higher frequency channels if your system is a hybrid system with analog channels utilizing the lower frequencies.

Jason Beck wrote a great writeup to the mythtv-users list on this topic. It's since been the defacto for getting MythTV set up to use your HDTV tuner card to tune QAM digital cable channels.

(Note that as of SVN 9627 and the 0.20 release, the channels.conf importer is no longer the default, in favor of the new channel scanner.)

Database Layout

There are two tables in the MythTV that we are concerned with, the channels table and the dtv_multiplex table. dtv_multiplex stores the frequencies for the channels, channels stores the actual channel information, with a reference to the dtv_multiplex table.

The best method for adding HDTV channels to the db is to populate channels with the channel information from your guide source (Schedules Direct, etc.), then modify the table to add the frequency, etc. information.

You will need to create a channel lineup at Schedules Direct containing the digital channels you intend to receive. Do not include analog channels nor any channels which you cannot or do not receive. You may have to go back and modify your lineup after the tests set out below.

Just remember that each channel lineup at Schedules Direct is a 'source' and will be associated with a 'sourceid' later on, and that each capture card gets associated with the particular lineup source that it can receive. Multiple cards can be associated with the same source. So make sure that your analog cards are associated with an all-analog lineup, and your digital cards with an all-digital lineup.

Scanning for channels

Warning.png Warning: the program 'dvbscan' on some systems is actually 'scan', if you try 'dvbscan' and get an error, try 'scan'

First a list of the receivable channels must be created. Be sure to have your hardware installed properly with the DVB drivers and firmware loading. Watch your /var/log/messages file for errors as your run these commands and address any problems you find.

Editor's Note: I have found MythTV's scanner to not do an adequate job of finding the appropriate channels. In order to ensure that all channels you are able to receive are found it is suggested to use the dvb-apps scan utility.
--Steveadeff 23:57, 1 February 2006 (UTC)

You must have already installed Dvb Apps. The relevant tools for this procedure include:

dvb-apps/util/scan/scan
dvb-apps/util/szap/azap

Run scan with the default table as an arguement, for open signals on a cable line, one of these two should work. HRC and IRC are different frequency methods that can be used, its hard to tell which your cable company uses without trying both. You may also not receive any channels this way, and may have to manually run the next step using all frequencies. More on this later. First, lets at least try atscscan. Note that atscscan does not automatically generate an output file. You probably want to redirect output when you begin this long process.

cd /usr/share/doc/dvb-utils/examples/scan 
or
cd dvb-apps/util/scan
./scan atsc/us-Cable-Standard-center-frequencies-QAM256

This process may take considerable time and report a lot of tuning failed messages. Let the entire process complete. If you do not find any channels with this freqency list, you can try some of the others like

us-Cable-HRC-center-frequencies-QAM256
us-Cable-IRC-center-frequencies-QAM256

You may also have to extend the frequencies if your cable provider uses higher frequencies than those in the file. To do this follow the increment, ie for HRC add 6 MegaHertz for each additional channel.

The output at the end will look similar to:

dumping lists (26 services)
KNTV-HD :207000000:QAM_256:65:68:4
NBC Wea :207000000:QAM_256:81:84:5
KBWB-HD :503000000:QAM_256:49:52:3
KBWB-SD :503000000:QAM_256:65:68:4
KGO-DT :533000000:QAM_256:49:52:3
KGO-DT :533000000:QAM_256:65:68:4
KTSF-D2:551000000:QAM_256:49:68:4
KTSF-D1:551000000:QAM_256:49:52:3
KPIX DT:563000000:QAM_256:49:52:1
KQED-HD:569000000:QAM_256:49:52:3
ENCORE:569000000:QAM_256:65:68:4
KMTP Digital Television:587000000:QAM_256:49:52:3
Telefutira:593000000:QAM_256:49:52:3
KCNS-DT:623000000:QAM_256:49:52:3
KKPX Faith:635000000:QAM_256:97:100:6
KKPX Worship:635000000:QAM_256:81:84:5
KKPX Pax East:635000000:QAM_256:65:68:4
KKPX Digital Television:635000000:QAM_256:49:52:3
KCSM-TV:647000000:QAM_256:33:36:2
Jazz-TV:647000000:QAM_256:49:52:3
KBHK-DT:659000000:QAM_256:49:52:1
KTVUHD:725000000:QAM_256:49:52:3
KTVU-DT:725000000:QAM_256:65:68:4
KRON-SD:731000000:QAM_256:49:52:3
KRON-HD:731000000:QAM_256:65:68:4
KTNC DIGITAL TELEVISION:767000000:QAM_256:49:52:3
Done.

You're lucky if you see channel names, it makes life easy, you may not, but thats ok, they can be figured out later.

If the output looks similar to the below output, please see DVB search for help.

[0006]:657000000:QAM_256:74:75:6
[000c]:657000000:QAM_256:0:80:12
[0001]:657000000:QAM_256:76:77:1
[0005]:663000000:QAM_256:0:65:5
[0006]:663000000:QAM_256:0:77:6
[0002]:663000000:QAM_256:0:68:2
[000b]:663000000:QAM_256:74:75:11
[000c]:663000000:QAM_256:72:73:12
.
.
[003d]:735000000:QAM_256:237:238:61
[0002]:741000000:QAM_256:0:72:2
[0006]:741000000:QAM_256:66:67:6
[0008]:741000000:QAM_256:74:75:8
[0009]:741000000:QAM_256:64:65:9
[000c]:741000000:QAM_256:0:78:12
[000a]:741000000:QAM_256:88:89:10

channels.conf Format

The format for each channel line in this list is as follows:

ChannelName:Frequency:Modulation:VID:AID:PID
  • Modulation should be QAM_256
  • VID is the decimal equivalent of the hex Video ID
  • AID is the decimal equivalent of the hex Audio ID
  • PID is the decimal equivalent of the hex Program ID

Important for the MythTV database are the frequency and Program ID.

If scan fails to find the Video ID you can run dvbtraffic to locate it. The video can easily be seen in dvbtraffic as it has > 1mbit bandwitdh and is usually AID -1 or AID -2.

Saving to channels.conf

Now you need to copy the lines with channel data (ignore the "dumping" and "Done." lines) to a file at $HOME/.azap/channels.conf so we can verify channels, and if needed figure out what they are.

Verifying channel tuning

Once this is done we can use the dvb-apps's azap utility to tune the different channels.

Running

./dvb-apps/util/szap/azap -r KNTV-HD

replacing KNTV-HD with the first name in your list should output similar to,

using '/dev/dvb/adapter0/frontend0' and '/dev/dvb/adapter0/demux0'
tuning to 207000000 Hz
video pid 0x0041, audio pid 0x0044
status 00 | signal 3999 | snr cd37 | ber 00000000 | unc 00000000 |
status 1f | signal cb84 | snr f8cf | ber 00000000 | unc 00000000 | FE_HAS_LOCK
status 1f | signal cb84 | snr f8b3 | ber 00000000 | unc 00000000 | FE_HAS_LOCK

Seeing FE_HAS_LOCK is what you are looking for, it means your card has succesfully tuned to the frequency. The channel however could be encrpyted so use mplayer to determine if the stream is playable.


It is a good idea to re-scan channels and create a list of channels which have some TV programs. My cable provider transmits signal on 300 channels but only 70 have some TV programs.

Process is like this:

Create script scan.sh:

mencoder  dvb://$1 -nosound -frames 10 -ovc lavc -o $1.avi &                                          
sleep 20                                                                                              
killall mencoder                                                                                      
sleep 1                                                                                               
killall -9 mencoder                                                                                   

Run following command, it will take a lot of time and produce avi files only for working channels:

cat ~/.mplayer/channels.conf | awk -F \: '{print $1}'  | xargs -i  scan.sh {}

You can review the files and delete those you are not interested in.

Create list of working channels:

find -name '*.avi' | sed 's/\.\///; s/\..*//' > list.of.working.channels

Create list of working channels for importing to myth:

join -t ':' list.of.working.channels ~/.mplayer/channels.conf > final.list


Import final.list into myth

Capturing frames with mplayer

Using another terminal we can watch the tuned channel using mplayer with

mplayer -nosound /dev/dvb/adapter0/dvr0

This will not play audio, its a safe way of testing, we won't have to worry about audio output, feel free to change this. If this fails, an alternative is

mplayer -nosound dvb://KNTV-HD

but you will need a copy of channels.conf in your .mplayer directory. This also works well if you are on a headless backend and wish to test your channels if your atscscan failed to pick up the channel names. With mplayer you can grab a JPG frame of the current channel using a command-line like this

mplayer -vo jpeg:quality=95 -frames 2 -ao null dvb://KNTV-HD

This will grab a single frame (mplayer has a bug which will not work with only frames 1) which you can then view quickly to determine the channel that was tuned.

Note: mplayer may have problems playing the audio in the HD stream due to the way ac3 audio is encoded, see http://www.pchdtv.com/forum/viewtopic.php?t=1048&start=0, but the stream itself may be fine.

Capturing frames with test_dvr

If for some reason your mplayer lacks the ability to tune DVB channels, you can also do this with the dvb-apps.

Run the azap command from before in a terminal window, note the video PID

using '/dev/dvb/adapter0/frontend0' and '/dev/dvb/adapter0/demux0'
tuning to 579000000 Hz
video pid 0x0840, audio pid 0x0841

Then, while the azap is running, in a second terminal window use test_dvr to capture the stream

BUFSIZE=1925120 ./test_dvr $HOME/test.mpg 0x0840

Break the capture using CTRL-C after a short period of time, then use mplayer to extract a frame

mplayer ~/test.mpg -vo jpeg:quality=95 -frames 2 -ao null


You can use these method to test all the channels in your list, make sure the names are correct, or figure out what channel it is if needed. I suggest creating a second channels.conf file and filling it with verified channels once you have tested each one.

Adding channels to the database

Now that we have a list of working channels, their frequency, and Program ID we are ready to begin entering the data into MythTV's database.

Automatically adding channels

Note: This process worked for me but I have not duplicated it as I have not tried destroying my working system to test it. If someone runs through these processes and finds it to be correct, please let me know

Adding a lineup on zap2it

First, I logged onto zap2it and added a second lineup for my HD programming that I have in my channels.conf file. In my case, this was covered by the Local Broadcast lineup but you may need to use a Digital Cable option if you have additional channels.

Importing channels.conf

Next, I went to the Input Connections screen, mapped my new input to my DVB card, and clicked on the Scan Channels button. Here I set my scan type to import and provided the path to my channels.conf file. Myth imported this information and created entries in my channel table.

You may need to add a Source before being able to import your channels.conf.

Populating program data

Next, I added this new lineup as HDTV in the Video Sources section of mythtv-setup on my master backend. Once this was completed, I ran mythfilldatabase and it downloaded all the program data for my channels as well as created the appropriate entires in the dtv_multiplex table and the channel table. The only thing that was missing was the proper mplexid and serviceid from my channels.conf.

For example "mythfilldatabase --sourceid 2 --do-channel-updates" may work for you if your new lineup is source 2.

Warning: I now had two sets of channels in my database, one from my import and one from mythfilldatabase. The channels from mythfilldatabase were missing the fields above. My imported channels had those fields

Fixup

In order to fix my channels, I noted the mplexid and serviceid from each channel that had been imported from channels.conf and then removed these channels from the database. I then inserted the noted mplexid and serviceid entries into the channel rows that mythfilldatabse created.

I ran mythfilldatabase again just to be sure and found that everything was in order.

Manually adding channels

dtv_multiplex

You may have noticed that many of your channels are on the same frequency with different Program ID's (and thus different Video and Audio ID's as well). This is because each frequency has the ability to carry many different streams of data, each is given their own id, with a program id for each channel referencing these video and audio id's. MythTV only needs the frequency and Program ID to properly tune a channel. The dtv_multiplex table is how MythTV keeps track of the frequencies for channels.

You can try using mythtv-setup's channel scan to try and find channels. Many find this won't work properly though. If things work well it should find all, or nearly all the channels you have in your custom channels.conf file. This will also have the added benefit of populating your dtv_multiplex table. If it doesn't work it may populate with non-existant channels and non-useful frequencies.

The final option is to import your custom channels.conf file. This won't work 100% properly and you will still have to edit the channels table, but it should give you a proper dtv_multiplex table.

It is suggested to modify or verify the table manually using [phpMyAdmin], a web based MySQL management tool. The table has 5 fields which we will find important,

  • mplexid is the individual id number for the frequency entry. Ideally they will be in numerical sequence, ie 1,2,3,4,5. If you 'Delete all capture cards' when you start, the id's will be in frequency order.
  • sourceid is the source for which the entry corresponds to, if your digital zap2it lineup is the second source you create in mythtv set, then this would be sourceid = 2. (Note that you can create your digital tuner in the Capture Card section first and its cardid will be '1', but it will be associated with sourceid =2.)
  • frequency is the frequency. Each frequency needs only be in the table once per source. So if you have 3 channels on the same frequency, it only needs to be in dtv_multiplex once for that source.
  • sistandard should be atsc, its default is dvb, so make sure its correct. Use 'update channel set sistandard = "atsc" where sourceid = 2;' to set this for all channels in sourceid 2.
  • modulation should be qam_256, for North American cable systems.

Now that dtv_multiplex is ready, print or write down what frequencies correspond to what mplexid, we will need these for the channel table.

channels

Now we can modify the channels table. Since we've already populated the table with the channel information for the guide, all we need to do is adjust the entries so that MythTV knows how to tune the channel. This only involves changing two fields, mplexid and serviceid.

In mysql use:

update channel set mplexid='x', serviceid='y' where channum='123';

To see what you are doing, try:

select chanid, channum, callsign, mplexid, serviceid from channel where sourceid='2';

(presuming that your HD3000 is sourceid 2). This will list the details for the 29 or whatever number of usable channels you receive.

You may be asking, "What about chanid, channum and freqid?". Well, none of these really matter when it comes to tuning the channel. chanid is the unique number MythTV gave that channel, it should be the sourceid followed by the channel number. freqid should be the same as channum which should be the channel your cable company assigned.

Now what you will need to do is use the information in your custom channels.conf file to enter the appropriate mplexid for the frequency of the channel from dtv_multiplex into mplexid. The other field, serviceid, is the programid number, the last number in the channel information line of channels.conf.

Lets Watch TV!

Once these two numbers are correct for all the channels associated with the HD tuner source you're ready to launch mythbackend and mythfrontend and watch or record some HD channels!

Share your results

If you have successfully executed this procedure, please post your channels.conf file on the Working QAM cable layouts page.

Troubleshooting

atscsrcid

If you are having trouble tuning channels that appear to be tuning properly with other application such as mplayer, check your channels table and make sure the atscsrcid column for any QAM channels is set to 0 (zero). These MPEG over QAM channels do not use the atscsrcid and having one populated can cause problems tuning.

Scan results

If you are having having trouble with this procedure because the initial scan returned alot of channels, and/or the channel names repeat see the following article DVB_search

Attenuators

If you have channel locks, but audio or video is choppy, you may be experiencing ghosting on the digital channels. This happens when there are multiple paths for the signals to your antenna and they each arrive at slightly different times. This is seen as ghosting on regular TV, but on digital TV, frames are dropped. The solution is to use an attenuator made specifically for this situation. I found a nice attenuator kit at http://www.sadoun.com/Sat/Products/S/20DB-Attenuator.htm.

Jason's Original Email

For posterity, I've left Jason Beck's orginal email here since its the guide that got most people going and inspired this article. I may also have made some mistakes in my attempt to clarify and add information to his process. This will allow anyone who finds the error to easily reference his work and fix my mistakes, though hopefully there are none ;-)--Steveadeff 23:57, 1 February 2006 (UTC)

http://www.gossamer-threads.com/lists/mythtv/users/150118?do=post_view_threaded#150118

If your provider is not kind enough to send the PSIP
info you will have to add the channels manually to the
database.

I'll try to explain how using you channels.conf as an
example. The database tables that need to be modified
are channel and dtv_multiplex. dtv_multiplex should
be populated by the channel scan and the channel table
you will need to modify yourself. Below is an example
of how you would setup your NBC (assuming
ch#707)channel in Myth.

1.) go to zap2it and build your listing for your local
HD broadcast channels.

2.) assign that listing to your dvb 3k in mythtvsetup

3.) go into the channel editor (still in mythtvsetup)
select your new listing and do a scan (select type
"cable")
The scan will take some time. Mostly likely you won't
really get anything useful, but what this will do it
populate the dtv_multiplex table in your db. You will
need this later on.

4.) exit mythtvsetup and run mythfilldatabase. This
will now provide you with your xmltvids to callsign
mappings.
At this point you still will not be able to tune
anything yet so don't try.

5.) ***Important*** stop mythbackend.

6.) edit your channels.conf and remove your PSIP
values. You don't need them for Myth. So your file
will look like this.
NBC:561000000:QAM_256:0:0
ABC:561000000:QAM_256:0:0
CBS:567000000:QAM_256:0:0
FOX:567000000:QAM_256:0:0

7.) go ahead and fire up azap and tune a channel.
$ azap -r NBC

8.) in a second terminial run dvbscan -c
$ dvbscan -c
What you need is the serviceid. This is the number in
the [000#] at the far left. Your provider will most
likely be using the same frequency for multiple
channels. So you may have more then one service per
freq. In this example your serviceid for NBC would be
2.
Example:
dumping lists (1 services)
[0002] (0x0002) 00: PCR == V V
0x0010 A 0x0011 (eng)
Now stop your dvbscan and azap tune.

9.) log into your database.
$mysql -u root -p mythconverg
To tune the 3k you will need to find the mplexid for
you known frequency.

10.) finding the mplexid
mysql> select mplexid from dtv_multiplex where
frequency = '561000000';
+---------+
| mplexid |
+---------+
| 25 |
+---------+
1 row in set (0.01 sec)
Write down the mplexid value. For example sake we
will say it was 25.

11.) Now you are ready to setup the channel. You have
the freqency (561000000),mplexid(25),and the
serviceid(2). Assuming that you now have a channel
707 from the previous mythfilldatabase above.
mysql> update channel set freqid="561000000" where
channum=707;
mysql> update channel set mplexid="25" where
channum=707;
mysql> update channel set serviceid="2" where
channum=707;
You may want to do a quick check to make sure all
looks good.
mysql> select * from channel where channum=707;
mysql> quit

12.) Start mythbackend

13.) Start up the frontend and test.

-- A couple of things I ran across later with the 3k.

By default the 3k will be set to record in PS mode. I
had wierd fastforward and rewind problems. In
mythtvsetup under capture cards-->dvb-->Advanced
settings I selected the "Use TS instead of PS" option
and the ffw and rew problems went away.

Also stopping the backend is only really needed to do
the azap tune. So while your there you may just want
to run the scan for serviceids for your 4 channels so
you only have to do it once.

I hope this helps. I just went through this about a
week ago and it was not easy to find all of the info.
Alot for trial and error. Let me know how it works
out for you...
-jason