Mar/100
Serialization to XML with CDATA tags
Sometimes you need CDATA tags around complex text in your destination XML.
There is no build in way of doing so (that I have found).
I found this nifty CdataWrapper class on Marc Gravell’s blog. Thanks Marc!
Here is the basic class.
public sealed class CDataWrapper : IXmlSerializable
{
// implicit to/from string
public static implicit operator string(CDataWrapper value)
{
return value == null ? null : value.Value;
}
public static implicit operator CDataWrapper(string value)
{
return value == null ? null : new CDataWrapper
{
Value =
value
};
}
public System.Xml.Schema.XmlSchema GetSchema()
{
return null;
}
// "" => <Node/>
// "Foo" => <Node><![CDATA[Foo]]></Node>
public void WriteXml(XmlWriter writer)
{
if (!string.IsNullOrEmpty(Value))
{
writer.WriteCData(Value);
}
}
// <Node/> => ""
// <Node></Node> => ""
// <Node>Foo</Node> => "Foo"
// <Node><![CDATA[Foo]]></Node> => "Foo"
public void ReadXml(XmlReader reader)
{
if (reader.IsEmptyElement)
{
Value = "";
}
else
{
reader.Read();
switch (reader.NodeType)
{
case XmlNodeType.EndElement:
Value = ""; // empty after all...
break;
case XmlNodeType.Text:
case XmlNodeType.CDATA:
Value = reader.ReadContentAsString();
break;
default:
throw new InvalidOperationException("Expected text/cdata");
}
}
}
// underlying value
public string Value { get; set; }
public override string ToString()
{
return Value;
}
}
To use it, you must change your DataMember from String to CDataWrapper, make it private and give it a public property.
This:
[DataMember]
pubblic string Answer { get; set; }
Becomes this:
pubblic string Answer { get; set; }
[DataMember(Name="Answer", EmitDefaultValue=false)]
private CDataWrapper AnswerCDATA
{
get { return Answer; }
set { Answer = value; }
}
Then the serialization is done with one line:
. . . XElement xml = Microsoft.ServiceModel.Web.SerializationExtensions.ToXml(object); .
So simple!!!!
Mar/100
More on Object Serialization to XML
It seems there are hundreds of ways to serialize object.
Here are just a few:
This one now seems very clumbsy. You pass in the object and the object type. It returns a string. I can’t see the point of using this one any more now that I have found the next one.
public static string ContractObjectToXml<T>(T obj)
{
DataContractSerializer dataContractSerializer = new DataContractSerializer(obj.GetType());
String text;
using (MemoryStream memoryStream = new MemoryStream())
{
dataContractSerializer.WriteObject(memoryStream, obj);
byte[] data = new byte[memoryStream.Length];
Array.Copy(memoryStream.GetBuffer(), data, data.Length);
text = Encoding.UTF8.GetString(data);
}
return text;
}
This one I have started using is the most simple I have seen so far.
. . . XElement xml = Microsoft.ServiceModel.Web.SerializationExtensions.ToXml(object); .
This one returns a stream. It gives you a bit more control over what is shown and included in the XML.
public static Stream Serialize(Object obj)
{
MemoryStream memoryStream = new MemoryStream();
DataContractSerializer dcSerializer = new DataContractSerializer(obj.GetType());
XmlWriterSettings xmlWriterSettings = new XmlWriterSettings();
xmlWriterSettings.Encoding = new UTF8Encoding(false);
xmlWriterSettings.ConformanceLevel = ConformanceLevel.Document;
xmlWriterSettings.Indent = true;
//xmlWriterSettings.OmitXmlDeclaration = true;
using (XmlWriter xWriter = XmlWriter.Create(memoryStream, xmlWriterSettings))
{
dcSerializer.WriteObject(xWriter, obj);
xWriter.Flush();
//return Encoding.UTF8.GetString(memoryStream.ToArray());
memoryStream.Position = 0;
return memoryStream;
}
}
Mar/100
SSIS package deployment and passwords
Something to remember when deploying your SSIS package:
The server will most likely run your package under a different user than the user you used to create the package. This causes a problem with the ‘ProtectionLevel’.
The default ProtectionLevel is EncryptSensitiveWithUserKey. During deployment, the user key will not be the same as yours. You need the same keys to encrypt and decrypt the connection strings and stuff like that.
This will give you the error: Failed to decrypt protected XML node “DTS:Password”
This error will only occure when you try to deploy, not while testing locally.
To fix this, change the ProtectionLevel to ‘EncryptSensitiveWithPassword’.
Then, when you create your job in the SQL Server Agent, you must enter your password.
This is done in the ‘Steps’ page:
– Under Type select SQL Server Integration Service Package
– Under Run as select SQL Server Agent Service Account
– Under the General tab and Package, select the path to your package (in the bid folder of your project or where ever you put it)
– Click Configurations or any other tab and you will then be prompted to enter your password in a popup.
Feb/100
GET and POST with JavaScript – with XMLHttpRequest
This is an example on how to make a POST call to a WCF service using JavaScript. Notice the XML to send is passed thru in the ’send’ method. The XML type must be sent if you are sending XML.
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<title></title>
<script type="text/javascript">
var url = 'http://example.com/myservice';
var xml = '<?xml version="1.0" encoding="utf-8" ?><Root xmlns="http://schemas.datacontract.org/2004/07/MyServiceDomain"><Name>John</Name><Language>UK</Langguage></Root>';
function getHTTPObject() {
if (typeof XMLHttpRequest != 'undefined') {
return new XMLHttpRequest();
}
try {
return new ActiveXObject("Msxml2.XMLHTTP");
} catch (e) {
try {
return new ActiveXObject("Microsoft.XMLHTTP");
} catch (e) { }
}
return false;
}
function useHttpResponse() {
if (http.readyState == 4) {
var textout = http.responseText;
doSomething(textout);
}
}
function doSomething(textout) {
alert(textout);
}
var http = getHTTPObject();
http.open("POST", url, true);
http.setRequestHeader("Content-type", "text/xml");
http.onreadystatechange = useHttpResponse;
http.send(xml);
</script>
</head>
<body>
</body>
</html>
NOTE: You cannot make calls to services or files outside your local domain. If you wish to call a service from another domain (for example a Yahoo or Google service), you must create a proxy page that calls the external service. Your JavaScript will call your proxy page.
Here is an example of a GET call. Notice that the URL and the request parameters are sent thru in the ‘open’ method. The ’send’ method has no arguments.
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<title></title>
<script type="text/javascript">
var url = 'http://example.com/myservice.php?name=John&Language=UK';
function getHTTPObject() {
if (typeof XMLHttpRequest != 'undefined') {
return new XMLHttpRequest();
}
try {
return new ActiveXObject("Msxml2.XMLHTTP");
} catch (e) {
try {
return new ActiveXObject("Microsoft.XMLHTTP");
} catch (e) { }
}
return false;
}
function useHttpResponse() {
if (http.readyState == 4) {
var textout = http.responseText;
doSomething(textout);
}
}
function doSomething(textout) {
alert(textout);
}
var http = getHTTPObject();
http.open("GET", url, true);
http.onreadystatechange = useHttpResponse;
http.send();
</script>
</head>
<body>
</body>
</html>
Feb/100
Temporary tables in SSIS
On one of our projects, we had some SQL that contained a temp table. This temp table is not one that will be used across several Data Flow Tasks. It is only used within a single one.
I don’t want to hear the arguments for not using temp tables. Sometimes you just have to. We found ourselves with one of those situations.
After a lot of research and reading of other people’s blog posts, I found a solution.
We had been using an OLE DB connection to our database.
Inside our Data Flow Task we placed our SQL inside an OLE DB Source.
None of the suggested solutions I read thru allowed me to use a temp table.
Our solution….
Use an ADO DB connection. Then an ADO NET Source.
Then under the properties of the ADO DB connection, we set
– RetainSameConnection = True
– DelayValidation = True
Most important, we didn’t use a temporary table (CREATE TABLE #Res …..). We used a variable table (DECLARE @Res TABLE …..) instead.
That did the trick.
I am glad this worked because I really didn’t want to deal with Global tables or defining a global table in the DB ahead of time or any workarounds like that. This was simple and it worked.
For some reason, this won’t work with an OLE DB connection. We think we know why, but have not found the documentation it.
Dec/090
Secure WCF REST webservice – part 1 – Validate XML with Schema
I have spent a good amount of time workig to implement a good XML validation method. I followed may leads and ended with a simple solution.
My goals were:
* use WCF/REST
* Use a Schema to validate the incoming XML request
* Extend the deserialize method turn prohibit DTDs and to disable external entities
* Filter IPs based on a list in the Config file
* Use the newer WebServiceHost2 from the WCF REST Starter Kit
Problems I had:
* I tried extending IDispatchMessageInspector and IClientMessageInspector to add schema validation. This worked great on WS Services (wsHttpBinding), but I could not get it to work on WebServices (webHttpBinding). I could not find any exampels of working solution on the net on how to do this with web services. (here is the blog post on what I did come up with). I abandoned this approach.
* Schema validation is done on the xml before it is derialized or when it is deserialized to an object. Because I could not access the deserialization before accepting the incoming xml request, I was forced to have the incoming object type ‘XElement’ rather than the destination object.
My solution:
* I used the new WCF REST Starter Kit. This reduces the confit file to almost nothing. The endpoints are set automatically. All adjustments can be done thru attributes on the service method itself.
Here is my config file:
<?xml version="1.0"?>
<configuration>
<appSettings>
...
</appSettings>
<system.web>
<compilation debug="true"/>
</system.web>
<system.serviceModel>
<serviceHostingEnvironment aspNetCompatibilityEnabled="true"/>
</system.serviceModel>
</configuration>
My Service:
// The following line sets the default namespace for DataContract serialized typed to be ""
[assembly: ContractNamespace("", ClrNamespace = "ThisIsMyService")]
namespace ThisIsMyService
{
// TODO: Please set IncludeExceptionDetailInFaults to false in production environments
[ServiceBehavior(IncludeExceptionDetailInFaults = true), AspNetCompatibilityRequirements(RequirementsMode = AspNetCompatibilityRequirementsMode.Allowed), ServiceContract]
public partial class Service
{
[WebHelp(Comment = "This is my service")]
[WebInvoke(UriTemplate = "MyService")]
[OperationContract]
public XElement MyService(XElement requestXml)
{
// Schema validation and deserialization are both done at the same time.
// If there is a problem, null is returned and an exception is thrown with a HTTPStatusCode of Bad Request.
var validatedObject = Helper.XmlToObject(requestXml, typeof(RequestObject),
ConfigurationManager.AppSettings["SchemaPath"].ToString(),
ConfigurationManager.AppSettings["SchemaNamespace"].ToString());
// I do all the process and other work in the helper class to keep this clean.
if (validatedObject is RequestObject)
return new Helper().DoStuff((RequestObject)validatedObject);
// If there is problem validating the incoming request XML, then the validated object will be null
// Error validating request - return nothing
return new XElement("root");
}
}
}
My service is very basic. All the logic is done in the helper class.
My helper class contains the static validation XMLtoObject method and the methods for doing the work of the service.
When you deserialize XML to an object, you can add a schema that will be used to check the XML.
What you don’t easily find in the documentation, is the fact that the Schema must be read and validated before it can be added to the reader that reads and deserializes the incoming xml request.
public class EngineServiceHelper
{
/// <summary>
/// A schema is an XML file like all the rest.
/// It also must be loaded and deserialized.
/// To use a schema, it must be placed in a XmlSchemaSet.
/// </summary>
/// <param name="xml"></param>
/// <param name="DestinationType"></param>
/// <param name="schemaPath"></param>
/// <param name="xmlNamespace"></param>
/// <returns></returns>
public static object XmlToObject(XElement xml, Type DestinationType, string schemaPath, string xmlNamespace)
{
ErrorMessage = string.Empty;
ErrorsCount = 0;
XmlSchemaSet schemaSet = new XmlSchemaSet();
schemaSet.ValidationEventHandler += new ValidationEventHandler(settings_ValidationEventHandler);
schemaSet.Add(xmlNamespace, schemaPath);
if (schemaSet.Count > 0)
{
try
{
// Set the setting for reading the Schema file
XmlReaderSettings schemaSettings = new XmlReaderSettings();
schemaSettings.ValidationType = ValidationType.Schema;
schemaSettings.Schemas.Add(schemaSet);
// To make the service more secure and prevent XML Bombs and such, disable DTDs
xmlRequestSettings.ProhibitDtd = true;
// Limit the size of the elements - 1024 = 1Kb
schemaSettings.MaxCharactersFromEntities = 1024;
// Adding the validation event handler prevents validation errors from throwing an exception.
// We want to control when and how an exception is thrown.
schemaSettings.ValidationEventHandler += new ValidationEventHandler(settings_ValidationEventHandler);
// create the reader for reading the schema
XmlReader schemaReader = XmlReader.Create(xml.CreateReader(), schemaSettings);
// Set the settings for reading the incoming XML request
XmlReaderSettings xmlRequestSettings = new XmlReaderSettings();
// To make the service more secure and prevent XML Bombs and such, disable DTDs
xmlRequestSettings.ProhibitDtd = true;
// Limit the size of the elements - 1024 = 1Kb
schemaSettings.MaxCharactersFromEntities = 1024;
// Create the reader for reading the XML request.
XmlReader xmlReader = XmlReader.Create(schemaReader, xmlRequestSettings);
// Deserialize the XML to an object
XmlSerializer serializer = new XmlSerializer(DestinationType, xmlNamespace);
object returnObject = serializer.Deserialize(xmlReader);
// If there are errors, throw an exception
if (ErrorsCount > 0)
throw new Exception(ErrorMessage);
// If ther are no problems, return the deserialized object
return returnObject;
}
catch (Exception ex)
{
// Using the WebProtocolException in the WCF REST Starter Kit, you can throw
// and exception and choose the HTTPStatusCode that will be displayed to the end user.
throw new WebProtocolException(HttpStatusCode.BadRequest, "Your XML is invalid: " + ex.Message, null);
}
}
return null;
}
/// <summary>
/// Check the error severity and create the error message.
/// Increment the error counter.
/// </summary>
/// <param name="sender"></param>
/// <param name="e"></param>
public static void settings_ValidationEventHandler(object sender, ValidationEventArgs e)
{
if (e.Severity == XmlSeverityType.Error)
{
ErrorMessage = "ERROR: " + ErrorMessage + e.Message + "\r\n";
ErrorsCount++;
}
else
{
ErrorMessage = "WARNING: " + ErrorMessage + e.Message + "\r\n";
ErrorsCount++;
}
}
}
The validation event handler allows you to catch the validation errors and handle them yourself.
I am checking the error severity and returning a message that reflects it.
Dec/090
Preventing XML Bombs
The Nobember (2009) issue of Msdn Magazine has a great article on XML Bombs and XML External Entity Attacks (DoS attatcks).
This was extremely useful to us as we are in the process of publishing a webservice that will be extensively used. Our current service is attacked regularly. Luckily, the attacks have been pretty basic and our security protects us. We were vulnerable to these types of attacks, but no longer are.
Example of an XML Bomb
XML Bombs use the XML document type definition (DTD) to create a piece of XML that, when parsed, will inflate to a huge size and consume all the resources on your machine.
Here is an example of such an attack from the article:
<?xml version="1.0"?> <!DOCTYPE lolz [ <!ENTITY lol "lol"> <!ENTITY lol2 "&lol;&lol;&lol;&lol;&lol;&lol;&lol;&lol;&lol;&lol;"> <!ENTITY lol3 "&lol2;&lol2;&lol2;&lol2;&lol2;&lol2;&lol2;&lol2;&lol2;&lol2;"> <!ENTITY lol4 "&lol3;&lol3;&lol3;&lol3;&lol3;&lol3;&lol3;&lol3;&lol3;&lol3;"> <!ENTITY lol5 "&lol4;&lol4;&lol4;&lol4;&lol4;&lol4;&lol4;&lol4;&lol4;&lol4;"> <!ENTITY lol6 "&lol5;&lol5;&lol5;&lol5;&lol5;&lol5;&lol5;&lol5;&lol5;&lol5;"> <!ENTITY lol7 "&lol6;&lol6;&lol6;&lol6;&lol6;&lol6;&lol6;&lol6;&lol6;&lol6;"> <!ENTITY lol8 "&lol7;&lol7;&lol7;&lol7;&lol7;&lol7;&lol7;&lol7;&lol7;&lol7;"> <!ENTITY lol9 "&lol8;&lol8;&lol8;&lol8;&lol8;&lol8;&lol8;&lol8;&lol8;&lol8;"> ]> <lolz>&lol9;</lolz>
The above code loops internally to create an object of about 3GB of memory. Impressive.
The 2nd entity consists of 10 of the 1st entity.
The 3rd entity consists of 10 of the 2nd entity (10 x 10 of the first).
The 4th entity consists of 10 of the 3rd entity (10 x 10 x 10 first).
And so on. The final XML is huge.
Preventing XML Bombs can be easily done by disabing inline DTDs.
Doing so is different in .Net 3.5 vs .Net 4.0
In .Net 3.5 you can use the ProhibitDtd boolean in the XmlTextReader or XmlReaderSettings.
// default is false XmlTextReader reader = new XmlTextReader(myStream); reader.ProhibitDtd = true; or // default is true XmlReaderSettings settings = new XmlReaderSettings(); settings.ProhibitDtd = true; XmlTextReader reader = new XmlTextReader(myStream, settings);
In .Net 4.0 you can use the DtdProcessing property. It can be set to Prohibit or Ingore. The ProhibitDtd property has been removed in 4.0.
// Prohibit will throw an exception if there is a <!DOCTYPE> XmlREaderSettings settings = new XmlReaderSettngs(); settings.DtdProcessing = DtdProcessing.Prohibit; XmlTextReader reader = new XmlTextReader(myStream, settings); or // Ingore will NOT throw an exception if there is a <!DOCTYPE>. It will ignore all DTDs. XmlREaderSettings settings = new XmlReaderSettngs(); settings.DtdProcessing = DtdProcessing.Prohibit; XmlTextReader reader = new XmlTextReader(myStream, settings);
If you would like to parse the DTDs, you should limit the size of the expanded DTDs.
This is easily done using the settings.
// default is true XmlReaderSettings settings = new XmlReaderSettings(); settings.ProhibitDtd = false; settings.MaxCharactersFromEntities = 1024; // 1 KB XmlTextReader reader = new XmlTextReader(myStream, settings);
Nov/090
Adding Multiple Namespaces to an XDocument/XElement
Lets say you want 3 namespaces added to your document.
You would define each one with the XNamespace property:
XNamespace defaultNS = "urn://xml.voodoo.net/vd/vd-1.0"; XNamespace a = "urn://xml.voodoo.net/vd/activity-1.0"; XNamespace f = "urn://xml.voodoo.net/vd/formating-1.0";
One of the namespaces will be your default.
When you create your rood element, you first add the default and then the other after.
The other 2 are added as you see below:
XElement rootElement = new XElement(defaultNS + "root", new XAttribute(XNamespace.Xmlns + "f", f.NamespaceName), new XAttribute(XNamespace.Xmlns + "a", a.NamespaceName));
To access the other namespaces, you
XElement somethingElement = new XElement(a + "something"); somethingElement.Value = "stuff and more stuff"; rootElement.Add(somethingElement);
The result will look like this:
<root xmlns:f="urn://xml.voodoo.net/vd/formating-1.0" xmlns:a="urn://xml.voodoo.net/vd/activity-1.0" xmlns="urn://xml.voodoo.net/vd/vd-1.0">
<a:something>stuff and more stuff</a:something>
</root>
Nov/090
Cloning an object – easy way
To clone an object MS says your object should inherit from IClonable. I have read many good and bad comments on this which mention it not being able to do a deel copy and whatever. A quick search will give you lots to read.
Using IClonable is a valid option.
Then I found one idea that I really liked. Serialize your object, than unserialize it. Done!
Things to note:
* Your object must be serializable.
* I have not tested if this works with complex object.
* I have not tested how this handles lazy bindings with nHibernate
* For simple object works great!
TODO: test with complex objects
Nov/090
Return a custom HTTP status code with a WCF service
We have a situation where we need to return a specific HTTP status code during some situations.
At first I thought I would have to use custom a FaultException. This looked promissing, but it was not required.
Then I thought perhaps I could use a fault Messages with the HttpResponseMessageProperty.
Then I foudn the easiest way was to set the WebOperationContext’s OutgoingResponse Status Code.
This was very simple.
private XElement ProcessQuestion(InputXMLObject inputXMLObject)
{
...
if (errorCode == "1.134")
{
WebOperationContext ctx = WebOperationContext.Current;
ctx.OutgoingResponse.StatusCode = System.Net.HttpStatusCode.SeeOther;
return new XElement("Error");
}
...
}
Setting the OutgoingResponse is all that is necessary to set the HTTP status code.
Very simple?