Building SageTV v9

[jusjoken]

Building the installer was made more difficult in that many of the needed files are not available today on GitHub as they need to be built first. I know that is a normal development process however unless I am missing something the build of everything needed for SageTV 9 is not ready right now on a windows machine or on a Linux machine.

The native parts of the windows app can be built now on VS2015 and the Sage.jar can be built on either windows or linux. Some of the thirdpary jars are stored in some of the thirdparty folders as well.

Building the remainder of the thirdparty apps seems to be possible under Linux as there are sh scripts in the build folder...but the release builds of these are not stored or kept on Git.

This means that to build the windows installer which needs ALL of these files I needed to grab the built versions from SageTV 7 install and make them available locally for the installer to grab. But that means others that change Windows builds would then need to locate these files as well.

Have I just missed something obvious to seasoned developers or what is the solution to this? Can we just store the release version of those thirdparty libraries on Git so they can be pulled down for inclusion? Do I zip them and make them available on BinTray to pull down as part of a setup of a build machine for windows installer or does someone (not me) need to create build scripts for these that run on windows so they can be built to be included in the installer?

Thoughts.

k

[Fuzzy]

Ultimately, everything that comes from source in SageTV's github, that is required to deploy the program, should be able to be built in a build script - ideally one that builds the program AND builds/packages the installer. Third party binaries that are obtained from other sources (lucene jar, for instance), I feel, should probably be pulled from their actual source by the build scripts for packaging. I'm pretty sure binaries built by source in google/sagetv should not then be added to google/sagetv.

[jusjoken]

Originally Posted by Fuzzy

Ultimately, everything that comes from source in SageTV's github, that is required to deploy the program, should be able to be built in a build script - ideally one that builds the program AND builds/packages the installer. Third party binaries that are obtained from other sources (lucene jar, for instance), I feel, should probably be pulled from their actual source by the build scripts for packaging. I'm pretty sure binaries built by source in google/sagetv should not then be added to google/sagetv.

I don't disagree, although lucene is one that is already built and stored on GitHub.

But in reality and in an OS community like this is that realistic? To build the install someone needs to have a java build environment, VS build for the natives and then a build setup for all the rest.

This is certainly beyond my capabilities or desires to put this type of thing together so if we are going to have a single build facility we need volunteers with those skills to pull it all together, script it and make it work well for other developers to work with as well.

k

[Fuzzy]

Originally Posted by jusjoken

I don't disagree, although lucene is one that is already built and stored on GitHub.

But in reality and in an OS community like this is that realistic? To build the install someone needs to have a java build environment, VS build for the natives and then a build setup for all the rest.

This is certainly beyond my capabilities or desires to put this type of thing together so if we are going to have a single build facility we need volunteers with those skills to pull it all together, script it and make it work well for other developers to work with as well.

k

I agree it would be quite a bit of work to get going, but the 'hard' part is making sure the scripts are robust enough to handle it. Then it's more a matter of specifying a design build environment for both Linux and Windows builds. With VS 2015 Community being free, I feel it's perfectly acceptable to leverage it as the build environment for windows. Beyond that, it all the vs project files, and linux build scripts are checked in to the git, than anyone who forks it and makes changes to the main sage source, would be able to build the complete package to a test installer (.deb, .tar.gz, .msi/exe) to try out on a test system, that probably wouldn't be their development machine.

[stuckless]

I think the windows side of things is going to take some time, because as you said, you need windows developers to make this happen. Sadly, I think that windows sort of expects you to be at the keyboard when you are doing builds. I know that people have said that is is possible, in theory, to build windows from an automated build system, but I haven't seen such a magical beast

The linux side is in better shape, mainly because Jeff wanted to have this running on Linux, and as such, the linux scripts build everything from end to end. And this nice, because as of now, when there is a pull request for SageTV the code is built and validated (for linux) on travis-ci. The next step here, is to then have the builds auto-deploy to bintray (which will likely happen this weekend).

But, back Windows...

While I think it's nice to rebuild EVERYTHING when something changes, I'm also OK with maintaining a set of binary artifacts that are only rebuilt when something changes that affect it. Dependency jars, like lucene, etc, is not something that we need/want to rebuild every time. So either we pull them from a binary repository (like maven) or we maintain them ourself. Currently we are doing both. For Lucene, we are pulling it from our "third_party" area, but for jogl, we are pulling it from maven. Eventually, I think all jars will be pulled from Maven, unless, they are not hosted there (ie, they are so old, they predate something like maven).

I think that eventually, Sage.jar and the default stv, should not require an updated installer, if that's all that changed. ie, Sage should update the Sage.jar itself, and the STV could be turned into a UI plugin. I think that an installer should only be re-created IF any of the native components changed. (this can be true for both linux and windows). In this way, unless you are changing the native code on window, you don't need to recreate the installer. This would simplify how often you need to rebuild the installer.

Now, when you are assembling a new windows installer, I don't think we should be taking anything from SageTV 7. ie, we should be using the SageTV 9 binaries. I think that people have created SageTV 9 binaries for all the core components (I think), just not sure if they are hosted on the BinTray site, or if they are hosting them elsewhere.

Maybe for the first couple of Windows installers, we need to scrape binaries from different sources, manually, but hopefully over time, we won't be rebuilding the installer just because the Sage.jar changed, or if we do, we basically just script the update to use the last set of native files (since they haven't changed), and we just update the Sage.jar and rebuild the installer.

[stuckless]

I did some searching, and AppVeyor appears to be a continous integration server for windows builds. It appears to be free, so, not sure if someone would like to take a stab at getting it setup. Maybe it could be used to validate and rebuild the windows components.

[Narflex]

The way that we did things at SageTV on Windows, was we had FFMPEG, MPlayer, libImageLoader and libFreetypeFont all as prebuilt binaries in our repository. Those files were not buildable under Visual Studio...and they also rarely changed. Everything else in the installer was rebuilt every time we did a new one since they could be rebuilt easily by standard tools. There's a lot more stuff to get setup properly if you want to be able to build these extra binaries above (MingW/msys). I would not be opposed to including these windows binaries in GitHub for this purpose. On Linux...they're easy to build...because you build everything with the same standard tools...that's why they get built every time there.

Of course ideally...they'll be rebuilt every time...but I don't want them to be an impediment to developers who aren't used to using GNU tools to build stuff on Windows.

[jusjoken]

I did a complete review for the 3 installers I am building (Server/Client/Placeshifter) and pulled a list of all the files I am currently getting from a SageTV7 install as the compiled version is not available to me (as far as I know).

ac3config.exe
ac3filter.ax
as_ir200l.dll
AUDIO_PREVIEW.dll
avcodec-51.dll
averapi_182.dll
avformat-51.dll
avutil-49.dll
CIAPI0.dll
CIAPI1.dll
CIAPI2.dll
CIAPI3.dll
CIAPI4.dll
CIAPI5.dll
Deinterlace.ax
DI_Adaptive.dll
DI_BlendedClip.dll
DI_Bob.dll
DI_EvenOnly.dll
DI_Greedy.dll
DI_Greedy2Frame.dll
DI_GreedyH.dll
DI_OddOnly.dll
DI_OldGame.dll
DI_ScalerBob.dll
DI_TwoFrame.dll
DI_VideoBob.dll
DI_VideoWeave.dll
DI_Weave.dll
dlcllib.dll
FixP4.dll
FreetypeFontJNI.dll
iconv.dll
ImageLoader.dll
jhead.exe
jpegtran.exe
ksProptyUtl.dll
Ma32.dll
Mase32.dll
Microsoft.VC80.CRT.manifest
MiniClient.jar
Mpeg2Transcoder.dll
msvcm80.dll
msvcp71.dll
msvcp80.dll
msvcr71.dll
msvcr80.dll
msvcrt.dll
mvp.bin
PsisDecd.dll
pushreader.dll
SageTVDX93D.dll
SageTVInfraredReceive.dll
SageTVPlaceshifter.exe
SageTVPlayer.exe
SageTVTranscoder.exe
SageTVWin32.dll
stvl2ad.ax
stvl2ae.ax
stvmcdsmpeg.ax
stvmcmpgdec.dll
stvmpeg2mux.ax
stvmpegin.dll
stvmpg_dlg.dll
stvmpgadec.dll
stvmpgaout.dll
stvmpgcap32.dll
stvmpgcheck.dll
stvmpgdmx.ax
stvmpgmux.dll
stvmpgvdec.dll
stvmpgvout.001
stvmpgvout.002
stvmpgvout.003
stvmpgvout.004
stvmpgvout.dll
stvmuxmpeg.ax
stvpcmaout.dll
swscale.dll
ttBdaDrvApi_Dll.dll
tvtvDLCL.dll
uu_irsage.dll
Win32ShellHook.dll
WinKeyboardHook.dll
WinRawInput.dll

I have also attached a CSV file which includes the folder the file is used in if that helps others review. The list is quite long but I would hope we can find a location to pull these files from to improve the build process for the installers.

Thanks
k

[Narflex]

I'll tell you what they all are as I recall.

ac3config.exe - part of AC3Filter (I'd suggest bundling the AC3Filter installer)
ac3filter.ax - part of AC3Filter (I'd suggest bundling the AC3Filter installer)
as_ir200l.dll - Plugin for the Actisys IR200L IR blaster (I'm fairly sure I pulled this from the SourceForge project for it)
AUDIO_PREVIEW.dll - Library for the Plextor PXTVU200 (or something like that) which is used for doing audio playback during live recording...usage of this was also disabled in the native libraries since we didn't know what license we could redistribute it under
avcodec-51.dll - FFMPEG library which I think is used by our StreamDemux
averapi_182.dll - Support for some specific AverMedia card, again disabled in the open source version due to licensing ambiguity
avformat-51.dll - FFMPEG library which I think is used by our StreamDemux
avutil-49.dll - FFMPEG library which I think is used by our StreamDemux
CIAPI0.dll - I think these are used by the Technotrend DVB cards
CIAPI1.dll
CIAPI2.dll
CIAPI3.dll
CIAPI4.dll
CIAPI5.dll
Deinterlace.ax - DScaler deinterlacing filter (I doubt anybody still uses this, it was popular when we started though and decoder filters didn't do a decent job of deinterlacing)
DI_Adaptive.dll - Libraries for the DScaler filter
DI_BlendedClip.dll
DI_Bob.dll
DI_EvenOnly.dll
DI_Greedy.dll
DI_Greedy2Frame.dll
DI_GreedyH.dll
DI_OddOnly.dll
DI_OldGame.dll
DI_ScalerBob.dll
DI_TwoFrame.dll
DI_VideoBob.dll
DI_VideoWeave.dll
DI_Weave.dll
dlcllib.dll - TVTV EPG data download file. I removed TVTV support from the open source version, nobody used it anymore AFAIK, so you don't need this.
FixP4.dll - Pentium 4 fix for the Provideo PVR256 card (the first card we supported, no one in their right mind would use this nowadays...it didn't even have hardware audio compression and A/V muxing leading to A/V sync issues sometimes)
FreetypeFontJNI.dll - SageTV native library for platform independent text rendering (uses FreeType), the source code is in native/crosslibs/Freetype
iconv.dll - This is an LGPL character conversion library...I have no idea what's using this and don't remember why its there.
ImageLoader.dll - SageTV image loading library (platform independent), source code is in third_party/SageTV-LGPL/imageload
jhead.exe - JHead executable, used for manipulating EXIF tags on JPEG files. This one has no license at all...their site says its free to use for whatever you want...and that's not a license, so we could not include it in our open source distribution
jpegtran.exe - This should get compiled as part of libjpeg. Source code is in third_party/codecs/jpeg-6b
ksProptyUtl.dll - Not sure...maybe this is a Hauppauge DLL? If so, then it can be put in the third_party/Hauppauge folder because we have their permission to redistribute their stuff. Although I don't see this file in my Windows install of SageTV.
Ma32.dll - I don't recall, and also don't see this in my Windows install of SageTV
Mase32.dll - I don't recall, and also don't see this in my Windows install of SageTV
Microsoft.VC80.CRT.manifest - Some kind of manifest file for the Microsoft DLLs that were included...don't recall why
MiniClient.jar - This is the JAR file for the SageTV Placeshifter build
Mpeg2Transcoder.dll - SageTV library for doing transmuxing of MP3 files for playback on the MediaMVP. Source is in native/so/Mpeg2Transcoder (source is for Linux...but we built it once for Windows years ago and never changed it again...now that I see this one; I *think* the FFMPEG dependencies above are for this DLL)
msvcm80.dll - Microsoft DLL for compatibility (not sure if needed anymore)
msvcp71.dll - Microsoft DLL for compatibility (not sure if needed anymore)
msvcp80.dll - Microsoft DLL for compatibility (not sure if needed anymore)
msvcr71.dll - Microsoft DLL for compatibility (not sure if needed anymore)
msvcr80.dll - Microsoft DLL for compatibility (not sure if needed anymore)
msvcrt.dll - Microsoft DLL for compatibility (not sure if needed anymore)
mvp.bin - Firmware for the MediaMVP. I'm not sure what the licensing is on this so we did not include it in the GitHub project. You can also download it from http://download.sagetv.com/mvp.bin
PsisDecd.dll - Microsoft library for doing MPEG2 PSI parsing...I don't recall exactly where this comes from...I think it was an MS redistributable.
pushreader.dll - SageTV native library for reading in push mode...although I don't recall when we ever actually used this. Source code is in third_party/SageTV-LGPL/Pushreader
SageTVDX93D.dll - SageTV JNI native library for Direct3D, source code is in native/dll/DX93D
SageTVInfraredReceive.dll - SageTV JNI native library for IR received, source code is in native/dll/InfraredReceive
SageTVPlaceshifter.exe - SageTV executable for the Placeshifter
SageTVPlayer.exe - Mplayer executable
SageTVTranscoder.exe - FFMPEG executable
SageTVWin32.dll - SageTV JNI native library for general OS stuff, source code is in native/dll/Win32DLL
stvl2ad.ax - Mainconcept MPEG2 audio decoder, commerically licensed, not redistributable
stvl2ae.ax - Mainconcept MPEG2 audio encoder, commerically licensed, not redistributable
stvmcdsmpeg.ax - Mainconcept MPEG2 demuxer, commerically licensed, not redistributable
stvmcmpgdec.dll - Mainconcept MPEG2 video decoder, commerically licensed, not redistributable
stvmpeg2mux.ax - Mainconcept MPEG2 multiplexor, commerically licensed, not redistributable
stvmpegin.dll - Mainconcept MPEG2 input library (never used), commerically licensed, not redistributable
stvmpg_dlg.dll - Mainconcept MPEG2 dialogs? (never used), commerically licensed, not redistributable
stvmpgadec.dll - Mainconcept MPEG2 audio decoder, commerically licensed, not redistributable
stvmpgaout.dll - Mainconcept MPEG2 audio output? (never used), commerically licensed, not redistributable
stvmpgcap32.dll - Mainconcept MPEG2 something or other (never used), commerically licensed, not redistributable
stvmpgcheck.dll - Mainconcept MPEG2 checker? (never used), commerically licensed, not redistributable
stvmpgdmx.ax - Mainconcept MPEG2 demux, commerically licensed, not redistributable
stvmpgmux.dll - Mainconcept MPEG2 mux, commerically licensed, not redistributable
stvmpgvdec.dll - Mainconcept MPEG2 video decoder, commerically licensed, not redistributable
stvmpgvout.001 - Mainconcept MPEG2 video encoder support file, commerically licensed, not redistributable
stvmpgvout.002 - Mainconcept MPEG2 video encoder support file, commerically licensed, not redistributable
stvmpgvout.003 - Mainconcept MPEG2 video encoder support file, commerically licensed, not redistributable
stvmpgvout.004 - Mainconcept MPEG2 video encoder support file, commerically licensed, not redistributable
stvmpgvout.dll - Mainconcept MPEG2 video encoder, commerically licensed, not redistributable
stvmuxmpeg.ax - Mainconcept MPEG2 mux, commerically licensed, not redistributable
stvpcmaout.dll - Mainconcept PCM audio handler, commerically licensed, not redistributable
swscale.dll - SWScaler library (I thought this was compiled into the image loader DLL; this may also be another FFMPEG dependency from the Mpeg2Transcoder). Source code is in third_party/swscale
ttBdaDrvApi_Dll.dll - Technotrend library for their cards (I didn't have the license info so redistribution was ambiguous)
tvtvDLCL.dll - TVTV file (again, we removed TVTV support in the open source version)
uu_irsage.dll - USBUIRT library, source code is in third_party/USB-UIRT
Win32ShellHook.dll - SageTV JNI library for Windows shell hooks, source code is in native/dll/Win32ShellHook
WinKeyboardHook.dll - SageTV JNI library for Windows keyboard hooks, source code is in native/dll/Win32KeyboardHook
WinRawInput.dll - SageTV JNI library for raw Windows input hooks (MCE remote), source code is in native/dll/WinRawInput

[Narflex]

That was a fun trip down memory lane.

[Fuzzy]

Originally Posted by Narflex

I'll tell you what they all are as I recall.

[A lot of great info]

This looks like a lot of things that should be depreciated out. Might be worth entering some issues in the tracker to remove things like DScaler and such, so references are not longer in the code for them, and so people who still might have it set up on upgrade don't get issues with it calling the filters that might not be there anymore.

[jusjoken]

Originally Posted by Narflex

That was a fun trip down memory lane.

I am almost sorry I asked

But seriously...thanks for the info....lots to figure out yet based on that list. I can clean up a bunch but will also need to chase down sources etc based on this info.

We will need to get some volunteers to build some of this source or I will need a crash course (or continue googling) to figure how all these things compile.

We will also need to get more people testing as we remove stuff that "might" not be needed

More fun ahead

k

[mechling-burgh]

Are those also the file that need to be recompiled to 64 bit support?

[Fuzzy]

Originally Posted by mechling-burgh

Are those also the file that need to be recompiled to 64 bit support?

They pretty much all would have to be if things stayed as they were, archetecturally. If we deprecated out a lot, it would simplify those issues. If a server-only was built, it would minimize the native binaries even further (essentially what would be required for recording and tuning). If it was limited to network encoders, it would be almost no native libraries.

I would like to retain the ability to make a 64-bit client, however, as I have a set of filters I would really like to use (www.svp-team.com), but they are huge memory hogs and simply cannot fit in the current Client's memory space. 64-bit directshow filters are limited in selection, though, and a 64-bit program cannot use 32-bit directshow, so it is not likely desired by many.

[JustFred]

Originally Posted by jusjoken

I did a complete review for the 3 installers I am building (Server/Client/Placeshifter) and pulled a list of all the files I am currently getting from a SageTV7 install as the compiled version is not available to me (as far as I know).

Several of the files in that list are built from our sources. Here's a list of everything built by vs2015:

DirecTVSerialControl.dll
DShowCapture.dll
DShowPlayer.dll
DShowTranscode.dll
MPEG2Dump.ax
MpegAudioEnc2.ax
MpegDeMux.ax
MpegMux.ax
MusicVisFilter.ax
PVR350OSDPlugin.dll
SageTV.exe
SageTVDX93D.dll
SageTVFileSrc.ax
SageTVInfraredReceive.dll
SageTVPlaceshifter.exe
SageTVServiceControl.exe
SageTVWin32.dll
StrmDeMux.ax
STVEVRPrstr.dll
TSSplitter.ax
TunerStub.dll
Win32ShellHook.dll
WinKeyboardHook.dll
WinRawInput.dll

I would have attached a .zip file with all those binaries, but it's size exceeds forum limitations.

[jusjoken]

Originally Posted by JustFred

Several of the files in that list are built from our sources. Here's a list of everything built by vs2015:

I would have attached a .zip file with all those binaries, but it's size exceeds forum limitations.

Ya, I should have pulled those off the list as I do build those from source prior to the installer build.

Still leaves a large list to deal with though.

k

[stuckless]

Originally Posted by Narflex

The way that we did things at SageTV on Windows, was we had FFMPEG, MPlayer, libImageLoader and libFreetypeFont all as prebuilt binaries in our repository. Those files were not buildable under Visual Studio...and they also rarely changed. Everything else in the installer was rebuilt every time we did a new one since they could be rebuilt easily by standard tools. There's a lot more stuff to get setup properly if you want to be able to build these extra binaries above (MingW/msys). I would not be opposed to including these windows binaries in GitHub for this purpose. On Linux...they're easy to build...because you build everything with the same standard tools...that's why they get built every time there.

Of course ideally...they'll be rebuilt every time...but I don't want them to be an impediment to developers who aren't used to using GNU tools to build stuff on Windows.

Jeff, are these able to be built under linux, targeting windows? Or, do they have to be built on a windows machine? If they can be rebuilt under linux, then I might, at some point, take a look at getting it working. If it requires windows... I'll leave for someone else

[SHS]

Originally Posted by Narflex

That was a fun trip down memory lane.

pthreadGC2.dll ??

Jeff I think the below dll are for Stream Machine SM2210 chip
Ma32.dll - Provideo PV256T, KWorld DVD Maker Possable even a AverMedia Card
Mase32.dll - Provideo PV256T, KWorld DVD Maker Possable even a AverMedia Card

[jusjoken]

I figured my first step would be to go through the list and REMOVE all the files identified as not used or not redistributable.

AUDIO_PREVIEW.dll
averapi_182.dll
Deinterlace.ax
dlcllib.dll
FixP4.dll
stvl2ad.ax
stvl2ae.ax
stvmcdsmpeg.ax
stvmcmpgdec.dll
stvmpeg2mux.ax
stvmpegin.dll
stvmpg_dlg.dll
stvmpgadec.dll
stvmpgaout.dll
stvmpgcap32.dll
stvmpgcheck.dll
stvmpgdmx.ax
stvmpgmux.dll
stvmpgvdec.dll
stvmpgvout.001
stvmpgvout.002
stvmpgvout.003
stvmpgvout.004
stvmpgvout.dll
stvmuxmpeg.ax
stvpcmaout.dll
tvtvDLCL.dll

After rebuilding the installer with these removed the client no longer plays any of my test videos (playback error).

Any suggestions?

k

[jusjoken]

Originally Posted by Narflex

ksProptyUtl.dll - Not sure...maybe this is a Hauppauge DLL? If so, then it can be put in the third_party/Hauppauge folder because we have their permission to redistribute their stuff. Although I don't see this file in my Windows install of SageTV.
Ma32.dll - I don't recall, and also don't see this in my Windows install of SageTV
Mase32.dll - I don't recall, and also don't see this in my Windows install of SageTV

These 3 files install into the Windows/SysWow64 folder when I install SageTV 7.

k